Bug 1318975
Summary: | AWS volumes remains in "in-use" status after deleting OSE pods which used them | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Jianwei Hou <jhou> |
Component: | Storage | Assignee: | Sami Wagiaalla <swagiaal> |
Status: | CLOSED ERRATA | QA Contact: | Jianwei Hou <jhou> |
Severity: | urgent | Docs Contact: | |
Priority: | urgent | ||
Version: | 3.2.0 | CC: | aos-bugs, bchilds, chaoyang, ekuric, eparis, jeder, jkaur, mmcgrath, swagiaal, tdawson |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | 1316095 | Environment: | |
Last Closed: | 2016-05-12 16:33:31 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1316095 | ||
Bug Blocks: |
Description
Jianwei Hou
2016-03-18 10:00:57 UTC
I saw it happen in Elvir's environment, unfortunately openshift does not log enough in these parts to find what's wrong. Trying hard to reproduce it with more logging, it's tedious (starting 50 pods takes a long time). I've tried on an OSE setup where the fix of https://github.com/openshift/ose/commit/27d9951039933065f416acac3a248eb39536ee5a is applied: openshift v3.1.1.6-29-g9a3b53e kubernetes v1.1.0-origin-1107-g4c8e6f4 etcd 2.1.2 I tried to create ebs volumes, pv, pvc, 20 pods, sleep 120, then delete these pods, pv, pvc and then create these pods again. Tried several times. Can not reproduce it. (In reply to Hou Jianwei from comment #2) > I've tried on an OSE setup where the fix of > https://github.com/openshift/ose/commit/ > 27d9951039933065f416acac3a248eb39536ee5a is applied: > > openshift v3.1.1.6-29-g9a3b53e > kubernetes v1.1.0-origin-1107-g4c8e6f4 > etcd 2.1.2 > > I tried to create ebs volumes, pv, pvc, 20 pods, sleep 120, then delete > these pods, pv, pvc and then create these pods again. Tried several times. > Can not reproduce it. Can you try to create more pods across more nodes, eg, try 40+ ( 50+ ) pods across 3 ( 4 ) nodes Elvir, Can you also try with the version Hou is using. The fix removing the cache (https://github.com/openshift/ose/commit/27d9951039933065f416acac3a248eb39536ee5a) introduced a lot of stability. Previously the cache could get out of sync and kubelet would not know which devices need to get detached. Elvir has confirmed that this bug cannot be reproduced with the latest version of origin. For OSE 3.1 please update https://bugzilla.redhat.com/show_bug.cgi?id=1316095 1. I create 26 ebs attach/detach the instance by aws cli command , and keep these ebs attach/detach 2. Create 26 pv, pvc, pods , sleep 120s, and delete these pods, pv and pvc, and create these pods again. Repeat step 2 several times Can not reproduced it openshift v3.2.0.7 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2016:1064 |