Bug 1977322
| Summary: | Unable to start instance due to stale volumes | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Paul Jany <pgodwin> | |
| Component: | openstack-cinder | Assignee: | Gorka Eguileor <geguileo> | |
| Status: | CLOSED ERRATA | QA Contact: | Evelina Shames <eshames> | |
| Severity: | high | Docs Contact: | Andy Stillman <astillma> | |
| Priority: | high | |||
| Version: | 16.1 (Train) | CC: | brian.rosmaita, eharney, eshames, geguileo, gkadam, jamsmith, jelynch, joflynn, ltoscano, lyarwood, pgodwin | |
| Target Milestone: | z9 | Keywords: | Triaged | |
| Target Release: | 16.1 (Train on RHEL 8.2) | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-cinder-15.4.0-1.20220707103428.58f0e73.el8ost | Doc Type: | Bug Fix | |
| Doc Text: |
Before this update, a race condition occurred when the Compute service (nova) requested the Block Storage service (cinder) to detach a volume and there was an external request to delete the volume. The race condition resulted in the volume failing to detach, the volume being deleted, and the Compute service being unable to remove the non-existent volume. With this update, the race condition is resolved.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 2057002 (view as bug list) | Environment: | ||
| Last Closed: | 2022-12-07 20:24:45 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 2057002 | |||
| Bug Blocks: | ||||
|
Description
Paul Jany
2021-06-29 13:14:04 UTC
After looking a the logs and Cinder code this seems to be a race condition with Cinder-CSI requests caused by an internal Cinder issue. Some context on the issue: - Cinder API uses the volume "status" field as a locking mechanism to prevent concurrent request processing on the same volume. - Most cinder operations are asynchronous, so the API returns before the operation has been completed by the cinder-volume service, but the attachment operations such as creating/updating/deleting an attachment are synchronous, so the API only returns to the caller after the cinder-volume service has completed the operation. - Our current code **incorrectly** modifies the status of the volume both on the cinder-volume and the cinder-api services on the attachment delete operation. The actual set of events that leads to the issue reported in this BZ are: [OpenShift] - Cinder-CSI requests Nova to detach volume (Request R1) [Nova] - R1: Asks cinder-api to delete the attachment and **waits** [Cinder-API] - R1: Checks the status of the volume - R1: Sends terminate connection request (R1) to cinder-volume and **waits** [Cinder-Volume] - R1: Ask the driver to terminate the connection - R1: The driver asks the backend to unmap and unexport the volume - R1: The status of the volume is changed in the DB to "available" [OpenShift] - Cinder-CSI asks Cinder to delete the volume (Request R2) [Cinder-API] - R2: Check that the volume's status is valid. It's available so it can be deleted. - R2: Tell cinder-volume to delete the volume and return immediately. [Cinder-Volume] - R2: Volume is deleted and DB entry is deleted - R1: Finish the termination of the connection [Cinder-API] - R1: Now that cinder-volume has finished the termination the code continues - R1: Try to modify the volume in the DB - R1: DB layer raises VolumeNotFound since the volume has been deleted from the DB - R1: VolumeNotFound is converted to HTTP 404 status code which is returned to Nova [Nova] - R1: Cinder responds with 404 on the attachment delete request - R1: Nova leaves the volume as attached, since the attachment delete failed At this point the Cinder and Nova DBs are out of sync, because Nova thinks that the attachment is connected and Cinder has detached the volume and even deleted it. **This is caused by a Cinder bug**, but there is some robustification work that could be done in Nova, since the volume could be left in a "detached from instance" state (since the os-brick call succeeded), and a second detach request could directly skip the os-brick call and when it sees that the volume or the attachment no longer exists in Cinder it can proceed to remove it from the instance's XML. This BZ will only focus on fixing the Cinder issue. I've raised https://github.com/kubernetes/cloud-provider-openstack/issues/1645 upstream as I honestly believe this is something that needs to be addressed in the cinder-csi plugin in k8s. Verification flow: - version: openstack-cinder-15.4.0-1.20221003203219.58f0e73.el8ost.noarch - Ceph backed deploymen - Booted an instance with attached volume (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+ | 5644fc51-1ae3-4ecc-b8a7-42ac077e6725 | in-use | Pansible_vol | 1 | tripleo | true | 64655a8f-72a3-4380-8960-c0c93c14dc48 | +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+ As this was already verified in a newer version, followed bug 2057002#c6 flow: delete loop in 3 parallel sessions: (overcloud) [stack@undercloud-0 ~]$ while true; do cinder delete 5621aeae-99c2-49c9-a740-c6e3f50bec79; done """ Delete for volume 5644fc51-1ae3-4ecc-b8a7-42ac077e6725 failed: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-50f74944-e015-4fa5-b4b4-dc19f5dd52d8) ERROR: Unable to delete any of the specified volumes. """ detach command: (overcloud) [stack@undercloud-0 ~]$ nova volume-detach inst1 5621aeae-99c2-49c9-a740-c6e3f50bec79 """ Request to delete volume 5644fc51-1ae3-4ecc-b8a7-42ac077e6725 has been accepted. Delete for volume 5644fc51-1ae3-4ecc-b8a7-42ac077e6725 failed: No volume with a name or ID of '5644fc51-1ae3-4ecc-b8a7-42ac077e6725' exists. ERROR: Unable to delete any of the specified volumes. """ Cinder list post deletion: (overcloud) [stack@undercloud-0 ~]$ cinder list +----+--------+------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +----+--------+------+------+-------------+----------+-------------+ +----+--------+------+------+-------------+----------+-------------+ I ran this same cycle several times, on all the cycles the volume was successfully detached followed by volume deletion as excepted. Moving to 'Verified'. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8795 |