Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1709288

Summary: [DR][MSTR-363] Recovery met error:failed to "StartContainer" for "controller-manager" with CrashLoopBackOff
Product: OpenShift Container Platform Reporter: zhou ying <yinzhou>
Component: MasterAssignee: Tomáš Nožička <tnozicka>
Status: CLOSED NOTABUG QA Contact: zhou ying <yinzhou>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.1.0CC: aos-bugs, jokerman, mfojtik, mmccomas, tnozicka, xxia
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-13 14:48:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description zhou ying 2019-05-13 10:39:09 UTC 
Description of problem:
Recovery failed with error: failed to "StartContainer" for "controller-manager" with CrashLoopBackOff

Version-Release number of selected component (if applicable):
[zhouying@dhcp-140-138 ~]$ oc version --short
Payload: 4.1.0-0.nightly-2019-05-09-204138

How reproducible:
Sometimes

Steps to Reproduce:
1. Fllow the Doc: https://docs.google.com/document/d/1ONkxdDmQVLBNJrSJymfKPrndo7b4vgCA2zwL9xHYx6A/edit to recovery for the cluster, and recovery succeed.
2. Update the base time to 1day in configmap: unsupported-cert-rotation-config
3. Force retate by delete the secrets and pods;
4. Wait for more than 20 mins, the cluster breaked.
5. Do the recovery steps for the master instance.


Actual results:
4. The cluster met error: x509: certificate has expired or is not yet valid
5. The cluster can't recovery back , see errors from master instance:

May 13 10:22:13 ip-172-31-131-67 hyperkube[1826]: E0513 10:22:13.311665    1826 pod_workers.go:190] Error syncing pod d142a459-7560-11e9-a836-024ff54832cc ("controller-manager-l2vgw_openshift-service-catalog-con
troller-manager(d142a459-7560-11e9-a836-024ff54832cc)"), skipping: failed to "StartContainer" for "controller-manager" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=controller-manager pod=con
troller-manager-l2vgw_openshift-service-catalog-controller-manager(d142a459-7560-11e9-a836-024ff54832cc)"
May 13 10:22:27 ip-172-31-131-67 hyperkube[1826]: E0513 10:22:27.813645    1826 remote_runtime.go:96] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sand
box k8s_packageserver-7d5d777b5d-m5rvj_openshift-operator-lifecycle-manager_fc087005-7568-11e9-bdce-024ff54832cc_0(cac1122e00dcb9df31cf5a6587f8c0c44e5537b5d12cebf504cd27f1a8b335b1): Multus: Err adding pod to net
work "openshift-sdn": cannot set "openshift-sdn" ifname to "eth0": no netns: failed to Statfs "/proc/85878/ns/net": no such file or directory
May 13 10:22:27 ip-172-31-131-67 hyperkube[1826]: E0513 10:22:27.813738    1826 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "packageserver-7d5d777b5d-m5rvj_openshift-operator-lifecycle-manager(fc087005-7
568-11e9-bdce-024ff54832cc)" failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_packageserver-7d5d777b5d-m5rvj_openshift-operator-lifecycle-manager_fc087005-7568-11e9-bdce-024ff548
32cc_0(cac1122e00dcb9df31cf5a6587f8c0c44e5537b5d12cebf504cd27f1a8b335b1): Multus: Err adding pod to network "openshift-sdn": cannot set "openshift-sdn" ifname to "eth0": no netns: failed to Statfs "/proc/85878/n
s/net": no such file or directory
May 13 10:22:27 ip-172-31-131-67 hyperkube[1826]: E0513 10:22:27.813757    1826 kuberuntime_manager.go:661] createPodSandbox for pod "packageserver-7d5d777b5d-m5rvj_openshift-operator-lifecycle-manager(fc087005-
7568-11e9-bdce-024ff54832cc)" failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_packageserver-7d5d777b5d-m5rvj_openshift-operator-lifecycle-manager_fc087005-7568-11e9-bdce-024ff54
832cc_0(cac1122e00dcb9df31cf5a6587f8c0c44e5537b5d12cebf504cd27f1a8b335b1): Multus: Err adding pod to network "openshift-sdn": cannot set "openshift-sdn" ifname to "eth0": no netns: failed to Statfs "/proc/85878/
ns/net": no such file or directory
May 13 10:22:27 ip-172-31-131-67 hyperkube[1826]: E0513 10:22:27.813845    1826 pod_workers.go:190] Error syncing pod fc087005-7568-11e9-bdce-024ff54832cc ("packageserver-7d5d777b5d-m5rvj_openshift-operator-life
cycle-manager(fc087005-7568-11e9-bdce-024ff54832cc)"), skipping: failed to "CreatePodSandbox" for "packageserver-7d5d777b5d-m5rvj_openshift-operator-lifecycle-manager(fc087005-7568-11e9-bdce-024ff54832cc)" with 
CreatePodSandboxError: "CreatePodSandbox for pod \"packageserver-7d5d777b5d-m5rvj_openshift-operator-lifecycle-manager(fc087005-7568-11e9-bdce-024ff54832cc)\" failed: rpc error: code = Unknown desc = failed to c
reate pod network sandbox k8s_packageserver-7d5d777b5d-m5rvj_openshift-operator-lifecycle-manager_fc087005-7568-11e9-bdce-024ff54832cc_0(cac1122e00dcb9df31cf5a6587f8c0c44e5537b5d12cebf504cd27f1a8b335b1): Multus:
 Err adding pod to network \"openshift-sdn\": cannot set \"openshift-sdn\" ifname to \"eth0\": no netns: failed to Statfs \"/proc/85878/ns/net\": no such file or directory"
May 13 10:22:28 ip-172-31-131-67 hyperkube[1826]: E0513 10:21:15.236079       1 controller_manager.go:303] Failed to get supported resources from server: Unauthorized
May 13 10:22:28 ip-172-31-131-67 hyperkube[1826]: E0513 10:21:15.236149       1 controller_manager.go:303] Failed to get supported resources from server: Unauthorized

Expected results:
4. The cluster should not be breaked;
5. Should recovery succeed.

Additional info: