Description of problem: co/kube-apiserver stuck in Progressing and Degraded after deleting openshift-kube-apiserver. Deleting project openshift-apiserver similarly, co/openshift-apiserver didn't hit such issue. Version-Release number of selected component (if applicable): 4.3.0-0.nightly-2019-11-21-230534 How reproducible: Always Steps to Reproduce: 1. Check below before deleting project openshift-kube-apiserver oc get po -n openshift-kube-apiserver -l apiserver --show-labels NAME READY STATUS RESTARTS AGE LABELS kube-apiserver-ip-10-0-134-189.ap-southeast-1.compute.internal 3/3 Running 0 101s apiserver=true,app=openshift-kube-apiserver,revision=13 kube-apiserver-ip-10-0-152-188.ap-southeast-1.compute.internal 3/3 Running 0 3m35s apiserver=true,app=openshift-kube-apiserver,revision=13 kube-apiserver-ip-10-0-175-115.ap-southeast-1.compute.internal 3/3 Running 0 5m26s apiserver=true,app=openshift-kube-apiserver,revision=13 oc get secret -n openshift-kube-apiserver | grep enc encryption-config Opaque 1 18m encryption-config-10 Opaque 1 140m encryption-config-11 Opaque 1 134m encryption-config-12 Opaque 1 123m encryption-config-13 Opaque 1 17m encryption-config-7 Opaque 1 5h4m encryption-config-8 Opaque 1 5h4m encryption-config-9 Opaque 1 4h56m 2. Delete project openshift-kube-apiserver # First remove all finalizers fields oc edit secret encryption-config encryption-config-{7..13} -n openshift-kube-apiserver secret/encryption-config edited ... secret/encryption-config-13 edited oc delete project openshift-kube-apiserver project.project.openshift.io "openshift-kube-apiserver" deleted 3. Check project Found it is Terminating oc get ns openshift-kube-apiserver -w NAME STATUS AGE openshift-kube-apiserver Terminating 6h8m oc get ns openshift-kube-apiserver -o yaml ... message: 'Some content in the namespace has finalizers remaining: encryption.apiserver.operator.openshift.io/deletion-protection in 2 resource instances' reason: SomeFinalizersRemain status: "True" type: NamespaceFinalizersRemaining Found new secrets in the Terminating project oc get secret -n openshift-kube-apiserver NAME TYPE DATA AGE encryption-config Opaque 1 2m8s encryption-config-14 Opaque 1 115s # Again, remove all finalizers fields oc edit secret -n openshift-kube-apiserver secret/encryption-config edited secret/encryption-config-14 edited 4. Check project oc get ns openshift-kube-apiserver openshift-kube-apiserver Active 2s oc get secret -n openshift-kube-apiserver | grep enc encryption-config Opaque 1 53s encryption-config-15 Opaque 1 33s 5. Check cluster state: oc get po -A | grep -vE "(Running|Completed)"; oc get co; oc get no ... kube-apiserver 4.3.0-0.nightly-2019-11-21-230534 True True True 7h37m ... Pods, nodes and other clusteroperators are well, only found co/kube-apiserver not well and stuck in Progressing and Degraded. oc describe co kube-apiserver ... Conditions: Last Transition Time: 2019-11-22T08:45:48Z Message: InstallerControllerDegraded: unable to set installer pod ownerrefs: configmap "revision-status-14" not found Reason: InstallerControllerDegradedError Status: True Type: Degraded Last Transition Time: 2019-11-22T08:43:42Z Message: Progressing: 3 nodes are at revision 13; 0 nodes have achieved new revision 16 Reason: Progressing Status: True Type: Progressing Check pods, still in revision=13 oc get po -n openshift-kube-apiserver -l apiserver --show-labels NAME READY STATUS RESTARTS AGE LABELS kube-apiserver-ip-10-0-134-189.ap-southeast-1.compute.internal 3/3 Running 0 93m apiserver=true,app=openshift-kube-apiserver,revision=13 kube-apiserver-ip-10-0-152-188.ap-southeast-1.compute.internal 3/3 Running 0 92m apiserver=true,app=openshift-kube-apiserver,revision=13 kube-apiserver-ip-10-0-175-115.ap-southeast-1.compute.internal 3/3 Running 0 92m apiserver=true,app=openshift-kube-apiserver,revision=13 Actual results: As above Expected results: co/kube-apiserver should not be stuck in Progressing and Degraded. Additional info:
Verified in 4.3.0-0.nightly-2019-12-13-180405 env: [xxia 2019-12-16 18:25:56 my]$ oc edit secret encryption-config encryption-config-{7..9} -n openshift-kube-apiserver secret/encryption-config edited secret/encryption-config-7 edited secret/encryption-config-8 edited secret/encryption-config-9 edited [xxia 2019-12-16 18:26:39 my]$ oc delete project openshift-kube-apiserver project.project.openshift.io "openshift-kube-apiserver" deleted [xxia 2019-12-16 18:26:49 my]$ oc get ns openshift-kube-apiserver -w Error from server (NotFound): namespaces "openshift-kube-apiserver" not found [xxia 2019-12-16 18:27:06 my]$ oc get ns openshift-kube-apiserver -w NAME STATUS AGE openshift-kube-apiserver Active 6s [xxia 2019-12-16 18:35:16 my]$ oc get ns openshift-kube-apiserver NAME STATUS AGE openshift-kube-apiserver Active 10m [xxia 2019-12-16 18:42:50 my]$ oc get co --no-headers | grep -v "True.*False.*False" # none
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062