Bug 1775580

Summary: [MSTR-485] co/kube-apiserver stuck in Progressing and Degraded after deleting openshift-kube-apiserver
Product: OpenShift Container Platform Reporter: Xingxing Xia <xxia>
Component: kube-apiserverAssignee: Luis Sanchez <sanchezl>
Status: CLOSED ERRATA QA Contact: Xingxing Xia <xxia>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.3.0CC: aos-bugs, mfojtik, sttts
Target Milestone: ---   
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1780809 (view as bug list) Environment:
Last Closed: 2020-01-23 11:13:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1780809    
Bug Blocks:    

Description Xingxing Xia 2019-11-22 10:52:43 UTC
Description of problem:
co/kube-apiserver stuck in Progressing and Degraded after deleting openshift-kube-apiserver. Deleting project openshift-apiserver similarly, co/openshift-apiserver didn't hit such issue.

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-11-21-230534

How reproducible:
Always

Steps to Reproduce:
1. Check below before deleting project openshift-kube-apiserver
oc get po -n openshift-kube-apiserver -l apiserver --show-labels
NAME                                                             READY   STATUS    RESTARTS   AGE     LABELS
kube-apiserver-ip-10-0-134-189.ap-southeast-1.compute.internal   3/3     Running   0          101s    apiserver=true,app=openshift-kube-apiserver,revision=13
kube-apiserver-ip-10-0-152-188.ap-southeast-1.compute.internal   3/3     Running   0          3m35s   apiserver=true,app=openshift-kube-apiserver,revision=13
kube-apiserver-ip-10-0-175-115.ap-southeast-1.compute.internal   3/3     Running   0          5m26s   apiserver=true,app=openshift-kube-apiserver,revision=13

oc get secret -n openshift-kube-apiserver | grep enc
encryption-config                               Opaque                                1      18m
encryption-config-10                            Opaque                                1      140m
encryption-config-11                            Opaque                                1      134m
encryption-config-12                            Opaque                                1      123m
encryption-config-13                            Opaque                                1      17m
encryption-config-7                             Opaque                                1      5h4m
encryption-config-8                             Opaque                                1      5h4m
encryption-config-9                             Opaque                                1      4h56m

2. Delete project openshift-kube-apiserver
# First remove all finalizers fields
oc edit secret encryption-config encryption-config-{7..13} -n openshift-kube-apiserver
secret/encryption-config edited
...
secret/encryption-config-13 edited

oc delete project openshift-kube-apiserver
project.project.openshift.io "openshift-kube-apiserver" deleted

3. Check project
Found it is Terminating
oc get ns openshift-kube-apiserver -w
NAME                       STATUS        AGE
openshift-kube-apiserver   Terminating   6h8m

oc get ns openshift-kube-apiserver -o yaml
...
    message: 'Some content in the namespace has finalizers remaining: encryption.apiserver.operator.openshift.io/deletion-protection
      in 2 resource instances'
    reason: SomeFinalizersRemain
    status: "True"
    type: NamespaceFinalizersRemaining

Found new secrets in the Terminating project
oc get secret -n openshift-kube-apiserver
NAME                   TYPE     DATA   AGE
encryption-config      Opaque   1      2m8s
encryption-config-14   Opaque   1      115s

# Again, remove all finalizers fields
oc edit secret -n openshift-kube-apiserver
secret/encryption-config edited
secret/encryption-config-14 edited

4. Check project
oc get ns openshift-kube-apiserver
openshift-kube-apiserver                                Active   2s

oc get secret -n openshift-kube-apiserver | grep enc
encryption-config                               Opaque                                1      53s
encryption-config-15                            Opaque                                1      33s

5. Check cluster state:
oc get po -A | grep -vE "(Running|Completed)"; oc get co; oc get no
...
kube-apiserver                             4.3.0-0.nightly-2019-11-21-230534   True        True          True       7h37m
...
Pods, nodes and other clusteroperators are well, only found co/kube-apiserver not well and stuck in Progressing and Degraded.

oc describe co kube-apiserver
...
  Conditions:
    Last Transition Time:  2019-11-22T08:45:48Z
    Message:               InstallerControllerDegraded: unable to set installer pod ownerrefs: configmap "revision-status-14" not found
    Reason:                InstallerControllerDegradedError
    Status:                True
    Type:                  Degraded
    Last Transition Time:  2019-11-22T08:43:42Z
    Message:               Progressing: 3 nodes are at revision 13; 0 nodes have achieved new revision 16
    Reason:                Progressing
    Status:                True
    Type:                  Progressing

Check pods, still in revision=13
oc get po -n openshift-kube-apiserver -l apiserver --show-labels
NAME                                                             READY   STATUS    RESTARTS   AGE   LABELS
kube-apiserver-ip-10-0-134-189.ap-southeast-1.compute.internal   3/3     Running   0          93m   apiserver=true,app=openshift-kube-apiserver,revision=13
kube-apiserver-ip-10-0-152-188.ap-southeast-1.compute.internal   3/3     Running   0          92m   apiserver=true,app=openshift-kube-apiserver,revision=13
kube-apiserver-ip-10-0-175-115.ap-southeast-1.compute.internal   3/3     Running   0          92m   apiserver=true,app=openshift-kube-apiserver,revision=13

Actual results:
As above

Expected results:
co/kube-apiserver should not be stuck in Progressing and Degraded.

Additional info:

Comment 3 Xingxing Xia 2019-12-16 10:45:16 UTC
Verified in 4.3.0-0.nightly-2019-12-13-180405 env:
[xxia 2019-12-16 18:25:56 my]$ oc edit secret encryption-config encryption-config-{7..9} -n openshift-kube-apiserver
secret/encryption-config edited
secret/encryption-config-7 edited
secret/encryption-config-8 edited
secret/encryption-config-9 edited
[xxia 2019-12-16 18:26:39 my]$ oc delete project openshift-kube-apiserver
project.project.openshift.io "openshift-kube-apiserver" deleted
[xxia 2019-12-16 18:26:49 my]$ oc get ns openshift-kube-apiserver -w
Error from server (NotFound): namespaces "openshift-kube-apiserver" not found
[xxia 2019-12-16 18:27:06 my]$ oc get ns openshift-kube-apiserver -w
NAME                       STATUS   AGE
openshift-kube-apiserver   Active   6s
[xxia 2019-12-16 18:35:16 my]$ oc get ns openshift-kube-apiserver
NAME                       STATUS   AGE
openshift-kube-apiserver   Active   10m

[xxia 2019-12-16 18:42:50 my]$ oc get co --no-headers | grep -v "True.*False.*False" # none

Comment 5 errata-xmlrpc 2020-01-23 11:13:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062