Bug 1626910
| Summary: | Pods and deploymntconfigs not deleted because of finalizer | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jaspreet Kaur <jkaur> | ||||||
| Component: | kube-controller-manager | Assignee: | Tomáš Nožička <tnozicka> | ||||||
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | zhou ying <yinzhou> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 3.9.0 | CC: | acai, aos-bugs, atripath, cshereme, erjones, fshaikh, gferrazs, jack.ottofaro, jkaur, jmalde, jokerman, jrfuller, mfojtik, misalunk, mmccomas, openshift-bugs-escalate, rbost, rekhan, rgregory, rkant, rvokal, scuppett, sttts, tnozicka | ||||||
| Target Milestone: | --- | Keywords: | Reopened | ||||||
| Target Release: | 3.9.z | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | |||||||||
| : | 1719240 (view as bug list) | Environment: | |||||||
| Last Closed: | 2019-08-09 09:10:10 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Can we get the following: 1) oc get pods -o wide 2) active openshift controller logs (at higher log level if possible) 3) the pods are stuck in terminating state - do we know why? we will probably need the node log where the pod is placed Can you also please elaborate more what are the exact steps they do? https://github.com/kubernetes/kubernetes/issues/65569#issuecomment-401430743 As per this depends on the DeleteOption setting, if you explicitly set it to "Orphan", then GC would not delete it. https://github.com/kubernetes/kubernetes/issues/65569 looks similar to this issue. Hi Michal,
So basically the issue that when we issue command to delete pod or a dc, a finalizer annotation gets added to them which avoid their deletion.
Scenarios ->
Issue scenario 1 ->
So in the environment when we see pod sometimes is terminating state for many days and then when we try to delete it using oc delete pod it does not delete it.(note grace period 0 also tried)
We saw the following in the finalizers annotation of the pod->
finalizers:
- foregroundDeletion
After deleting the above annotation the pod gets deleted.
Issue scenario 2 ->
When we try to delete dc using oc delete dc, it does not get deleted.
In deployment config, we see the following finalizers annotation.
apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
annotations:
openshift.io/generated-by: OpenShiftNewApp
creationTimestamp: 2018-09-07T02:22:25Z
deletionGracePeriodSeconds: 0
deletionTimestamp: 2018-09-10T02:20:28Z
finalizers:
- orphan
After removing the following from the dc
finalizers:
- orphan
the dc immediately gets deleted.
Now the main concern is why after issuing a delete on the pod or dc resource does not delete it and that why we are manually needing to remove the finalizers from the annotation to have them deleted.
Thanks and regards,
Miheer Salunke.
OK can you let us know in which version of OpenShift we have the fix ? Also can how can we confirm we are hitting https://github.com/kubernetes/kubernetes/issues/56348 ? Any specific logs having this issue info ? `oc version` ? Created attachment 1589230 [details]
master api/controllers journal logs
Created attachment 1589231 [details]
etcd journal logs
a month without without providing the requested information, likely not a bug The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |
Description of problem : When deleting pods it stucks in terminating state and similarly the deployment configs is not deleted. In pods we see : finalizers: - foregroundDeletion After which the pod is deleted and in deplpoyment config, below annotation prevents deletion: apiVersion: apps.openshift.io/v1 kind: DeploymentConfig metadata: annotations: openshift.io/generated-by: OpenShiftNewApp creationTimestamp: 2018-09-07T02:22:25Z deletionGracePeriodSeconds: 0 deletionTimestamp: 2018-09-10T02:20:28Z finalizers: - orphan Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Fails to delete openshift objects Expected results: There should be finalizer added which prevents deletion. Additional info: