Bug 1775580
| Summary: | [MSTR-485] co/kube-apiserver stuck in Progressing and Degraded after deleting openshift-kube-apiserver | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Xingxing Xia <xxia> | |
| Component: | kube-apiserver | Assignee: | Luis Sanchez <sanchezl> | |
| Status: | CLOSED ERRATA | QA Contact: | Xingxing Xia <xxia> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 4.3.0 | CC: | aos-bugs, mfojtik, sttts | |
| Target Milestone: | --- | |||
| Target Release: | 4.3.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1780809 (view as bug list) | Environment: | ||
| Last Closed: | 2020-01-23 11:13:47 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1780809 | |||
| Bug Blocks: | ||||
Verified in 4.3.0-0.nightly-2019-12-13-180405 env:
[xxia 2019-12-16 18:25:56 my]$ oc edit secret encryption-config encryption-config-{7..9} -n openshift-kube-apiserver
secret/encryption-config edited
secret/encryption-config-7 edited
secret/encryption-config-8 edited
secret/encryption-config-9 edited
[xxia 2019-12-16 18:26:39 my]$ oc delete project openshift-kube-apiserver
project.project.openshift.io "openshift-kube-apiserver" deleted
[xxia 2019-12-16 18:26:49 my]$ oc get ns openshift-kube-apiserver -w
Error from server (NotFound): namespaces "openshift-kube-apiserver" not found
[xxia 2019-12-16 18:27:06 my]$ oc get ns openshift-kube-apiserver -w
NAME STATUS AGE
openshift-kube-apiserver Active 6s
[xxia 2019-12-16 18:35:16 my]$ oc get ns openshift-kube-apiserver
NAME STATUS AGE
openshift-kube-apiserver Active 10m
[xxia 2019-12-16 18:42:50 my]$ oc get co --no-headers | grep -v "True.*False.*False" # none
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062 |
Description of problem: co/kube-apiserver stuck in Progressing and Degraded after deleting openshift-kube-apiserver. Deleting project openshift-apiserver similarly, co/openshift-apiserver didn't hit such issue. Version-Release number of selected component (if applicable): 4.3.0-0.nightly-2019-11-21-230534 How reproducible: Always Steps to Reproduce: 1. Check below before deleting project openshift-kube-apiserver oc get po -n openshift-kube-apiserver -l apiserver --show-labels NAME READY STATUS RESTARTS AGE LABELS kube-apiserver-ip-10-0-134-189.ap-southeast-1.compute.internal 3/3 Running 0 101s apiserver=true,app=openshift-kube-apiserver,revision=13 kube-apiserver-ip-10-0-152-188.ap-southeast-1.compute.internal 3/3 Running 0 3m35s apiserver=true,app=openshift-kube-apiserver,revision=13 kube-apiserver-ip-10-0-175-115.ap-southeast-1.compute.internal 3/3 Running 0 5m26s apiserver=true,app=openshift-kube-apiserver,revision=13 oc get secret -n openshift-kube-apiserver | grep enc encryption-config Opaque 1 18m encryption-config-10 Opaque 1 140m encryption-config-11 Opaque 1 134m encryption-config-12 Opaque 1 123m encryption-config-13 Opaque 1 17m encryption-config-7 Opaque 1 5h4m encryption-config-8 Opaque 1 5h4m encryption-config-9 Opaque 1 4h56m 2. Delete project openshift-kube-apiserver # First remove all finalizers fields oc edit secret encryption-config encryption-config-{7..13} -n openshift-kube-apiserver secret/encryption-config edited ... secret/encryption-config-13 edited oc delete project openshift-kube-apiserver project.project.openshift.io "openshift-kube-apiserver" deleted 3. Check project Found it is Terminating oc get ns openshift-kube-apiserver -w NAME STATUS AGE openshift-kube-apiserver Terminating 6h8m oc get ns openshift-kube-apiserver -o yaml ... message: 'Some content in the namespace has finalizers remaining: encryption.apiserver.operator.openshift.io/deletion-protection in 2 resource instances' reason: SomeFinalizersRemain status: "True" type: NamespaceFinalizersRemaining Found new secrets in the Terminating project oc get secret -n openshift-kube-apiserver NAME TYPE DATA AGE encryption-config Opaque 1 2m8s encryption-config-14 Opaque 1 115s # Again, remove all finalizers fields oc edit secret -n openshift-kube-apiserver secret/encryption-config edited secret/encryption-config-14 edited 4. Check project oc get ns openshift-kube-apiserver openshift-kube-apiserver Active 2s oc get secret -n openshift-kube-apiserver | grep enc encryption-config Opaque 1 53s encryption-config-15 Opaque 1 33s 5. Check cluster state: oc get po -A | grep -vE "(Running|Completed)"; oc get co; oc get no ... kube-apiserver 4.3.0-0.nightly-2019-11-21-230534 True True True 7h37m ... Pods, nodes and other clusteroperators are well, only found co/kube-apiserver not well and stuck in Progressing and Degraded. oc describe co kube-apiserver ... Conditions: Last Transition Time: 2019-11-22T08:45:48Z Message: InstallerControllerDegraded: unable to set installer pod ownerrefs: configmap "revision-status-14" not found Reason: InstallerControllerDegradedError Status: True Type: Degraded Last Transition Time: 2019-11-22T08:43:42Z Message: Progressing: 3 nodes are at revision 13; 0 nodes have achieved new revision 16 Reason: Progressing Status: True Type: Progressing Check pods, still in revision=13 oc get po -n openshift-kube-apiserver -l apiserver --show-labels NAME READY STATUS RESTARTS AGE LABELS kube-apiserver-ip-10-0-134-189.ap-southeast-1.compute.internal 3/3 Running 0 93m apiserver=true,app=openshift-kube-apiserver,revision=13 kube-apiserver-ip-10-0-152-188.ap-southeast-1.compute.internal 3/3 Running 0 92m apiserver=true,app=openshift-kube-apiserver,revision=13 kube-apiserver-ip-10-0-175-115.ap-southeast-1.compute.internal 3/3 Running 0 92m apiserver=true,app=openshift-kube-apiserver,revision=13 Actual results: As above Expected results: co/kube-apiserver should not be stuck in Progressing and Degraded. Additional info: