Bug 1737788 - CSI volumes are not detached on RequestLimitExceeded errors
Summary: CSI volumes are not detached on RequestLimitExceeded errors
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.2.0
Assignee: Jan Safranek
QA Contact: Chao Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-06 08:42 UTC by Jan Safranek
Modified: 2019-10-16 06:35 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-16 06:34:52 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift csi-external-attacher pull 16 0 None closed Bug 1737788: UPSTREAM: 169: Fixed ControllerUnpublish error handling 2020-12-09 16:42:07 UTC
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:35:01 UTC

Description Jan Safranek 2019-08-06 08:42:42 UTC
Description of problem:
When AWS EBS CSI driver returns "RequestLimitExceeded" to ControllerUnpublishRequest (="detach"), csi-external-attacher marks the volume as detached:

I0805 16:05:08.948172       1 connection.go:180] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0805 16:05:08.948182       1 connection.go:181] GRPC request: {"node_id":"i-0eb51d1f90e270d91","volume_id":"vol-0766d9cc3c1966b86"}
I0805 16:05:15.323947       1 connection.go:183] GRPC response: {}
I0805 16:05:15.324406       1 connection.go:184] GRPC error: rpc error: code = Internal desc = Could not detach volume "vol-0766d9cc3c1966b86" from node "i-0eb51d1f90e270d91": could not detach volume "vol-0766d9cc3c1966b86" from node "i-0eb51d1f90e270d91": RequestLimitExceeded: Request limit exceeded.
        status code: 503, request id: 10ceab6c-4a6d-4da5-add2-d46d4fdb652a
I0805 16:05:15.324422       1 csi_handler.go:369] Detached "csi-51ad5c68abe99844c08cfce659c0f8375d8b0255391a6b28b543501577b3dee6" with error rpc error: code = Internal desc = Could not detach volume "vol-0766d9cc3c1966b86" from node "i-0eb51d1f90e270d91": could not detach volume "vol-0766d9cc3c1966b86" from node "i-0
eb51d1f90e270d91": RequestLimitExceeded: Request limit exceeded.
        status code: 503, request id: 10ceab6c-4a6d-4da5-add2-d46d4fdb652a
I0805 16:05:15.324459       1 util.go:70] Marking as detached "csi-51ad5c68abe99844c08cfce659c0f8375d8b0255391a6b28b543501577b3dee6"

The volume remains attached to a node and the external-attacher never re-tries to detach the volume.

Version-Release number of selected component (if applicable):
4.2.0-0.okd-2019-08-05-143844

How reproducible:
rarely

Steps to Reproduce:
1. run a pod with AWS EBS volume provided by CSI.
2. delete the pod and hope for RequestLimitExceeded response.

Actual results:
1. The volume is still attached to the node (as seen in AWS console)
2. VolumeAttachment Kubernetes object is deleted

Expected results:
1. The volume is detached from the node (after a while).
2. VolumeAttachment Kubernetes object is deleted.

Comment 1 Jan Safranek 2019-08-06 08:43:21 UTC
Filed https://github.com/kubernetes-csi/external-attacher/pull/165

Comment 4 Chao Yang 2019-08-21 02:47:44 UTC
Created 400 pvc, volumes and pods, but did not meet the AWS "RequestLimitExceeded".
Mark this bug to be verified. Will re-open it if this issue happen again.

Comment 5 errata-xmlrpc 2019-10-16 06:34:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.