Bug 1548628
Summary: | Displayed "pv deleted failed" when deleted projects but ebs volume was deleted indeed | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Chao Yang <chaoyang> | ||||||
Component: | Storage | Assignee: | Jan Safranek <jsafrane> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Chao Yang <chaoyang> | ||||||
Severity: | medium | Docs Contact: | |||||||
Priority: | medium | ||||||||
Version: | 3.9.0 | CC: | aos-bugs, aos-storage-staff, bchilds, jsafrane, wsun | ||||||
Target Milestone: | --- | ||||||||
Target Release: | 3.10.0 | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | No Doc Update | |||||||
Doc Text: |
undefined
|
Story Points: | --- | ||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2018-07-30 19:09:51 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Description
Chao Yang
2018-02-24 03:51:28 UTC
Created attachment 1400103 [details]
node log
Fom the log it seems that something (old controller?) deleted EBS volume and did not delete associated PV. New controller failed to delete the volume because it did not exist. Cloud provider should be idempotent and report success in this case. Feb 23 10:58:00 ip-172-18-14-168 atomic-openshift-master-controllers: E0223 10:58:00.751176 123899 aws.go:2225] Error describing volume "vol-085a251610877ae5d": "error querying ec2 for volume \"vol-085a251610877ae5d\": \"error listing AWS volumes: \\\"InvalidVolume.NotFound: The volume 'vol-085a251610877ae5d' does not exist.\\\\n\\\\tstatus code: 400, request id: cb327c80-9f10-4462-905b-8c75878fb86c\\\"\"" Feb 23 10:58:00 ip-172-18-14-168 atomic-openshift-master-controllers: E0223 10:58:00.751192 123899 aws.go:2211] error querying ec2 for volume "vol-085a251610877ae5d": "error listing AWS volumes: \"InvalidVolume.NotFound: The volume 'vol-085a251610877ae5d' does not exist.\\n\\tstatus code: 400, request id: cb327c80-9f10-4462-905b-8c75878fb86c\"" Feb 23 10:58:00 ip-172-18-14-168 atomic-openshift-master-controllers: I0223 10:58:00.751200 123899 aws_util.go:57] Error deleting EBS Disk volume aws://us-east-1d/vol-085a251610877ae5d: error querying ec2 for volume "vol-085a251610877ae5d": "error listing AWS volumes: \"InvalidVolume.NotFound: The volume 'vol-085a251610877ae5d' does not exist.\\n\\tstatus code: 400, request id: cb327c80-9f10-4462-905b-8c75878fb86c\"" Upstream PR: https://github.com/kubernetes/kubernetes/pull/60490 Hi Jan, Tested this issue as below steps: 1.Create a new project named 8hdv3 2.Create 10 dynamic pvc 3.Create 10 pods 4.After 10 pods are running, restart atomic-openshift-master-controllers services 3 times. 5.Delete project I found the pv entered into failed status for some seconds and the were deleted. Not sure it is not right or by design. Could you help double confirm? [root@ip-172-18-9-130 ~]# oc describe pv pvc-a4b2509e-58e8-11e8-9cbc-0e566b69876a Name: pvc-a4b2509e-58e8-11e8-9cbc-0e566b69876a Labels: failure-domain.beta.kubernetes.io/region=us-east-1 failure-domain.beta.kubernetes.io/zone=us-east-1d Annotations: kubernetes.io/createdby=aws-ebs-dynamic-provisioner pv.kubernetes.io/bound-by-controller=yes pv.kubernetes.io/provisioned-by=kubernetes.io/aws-ebs Finalizers: [kubernetes.io/pv-protection] StorageClass: gp2 Status: Failed Claim: 2yow9/dynamic-pvc-1 Reclaim Policy: Delete Access Modes: RWO Capacity: 1Gi Node Affinity: <none> Message: Error deleting EBS volume "vol-0c692a1ffaa4fa50b" since volume is currently attached to "i-0d92bac96c2736e25" Source: Type: AWSElasticBlockStore (a Persistent Disk resource in AWS) VolumeID: aws://us-east-1d/vol-0c692a1ffaa4fa50b FSType: ext4 Partition: 0 ReadOnly: false Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning VolumeFailedDelete 1m persistentvolume-controller Error deleting EBS volume "vol-0c692a1ffaa4fa50b" since volume is currently attached to "i-0d92bac96c2736e25" > I found the pv entered into failed status for some seconds and the were deleted.
This is ok and expected, as long as the PV is deleted in couple of seconds (say 1 minute). It would be a bug if it stays this way forever.
It is passed. Thanks for Jan's double confirm. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1816 |