Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1737630

Summary:	Shutdown Clusters don't come back up
Product:	OpenShift Container Platform	Reporter:	Wolfgang Kulhanek <wkulhane>
Component:	kube-apiserver	Assignee:	David Eads <deads>
Status:	CLOSED ERRATA	QA Contact:	zhou ying <yinzhou>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.1.z	CC:	aos-bugs, deads, mfojtik, nagrawal, xxia, yinzhou
Target Milestone:	---
Target Release:	4.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-01-23 11:05:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Wolfgang Kulhanek 2019-08-05 21:40:33 UTC

Description of problem:
When shutting down a cluster (all VMs that make up the cluster) before the 24h bootstrap cert rotation (??) the cluster fails to start back up after the 24h period.

We absolutely positively need a way to shut down clusters shortly (8h) after installation to save Red Hat money.

Version-Release number of selected component (if applicable):
4.1.8


How reproducible:
Every time

Steps to Reproduce:
1. Install an OCP 4 Cluster
2. Stop the VMs after the installation is complete
3. 26h later resume the VMs

Actual results:
All nodes (Masters and Workers) become NotReady with no way to recover.

@deads2k suggested deploying a DaemonSet to fix this - but this DaemonSet doesn't seem to work. Nodes issue CSRs - which I can manually approve. But the certificates never get issued - so every 12 minutes the nodes issue another CSR.

I also tried following the documentation (https://docs.openshift.com/container-platform/4.1/disaster_recovery/scenario-3-expired-certs.html) but was not able to get that to work either. It ends at the same result. Nodes are in NotReady and nothing will accept the approved CSRs

Expected results:
Cluster resumes fine.

Additional info:

The DaemonSet that David Eads suggested is here:
https://github.com/redhat-cop/agnosticd/blob/development/ansible/roles/ocp4-workload-enable-cluster-shutdown/files/daemon_set.yaml

Comment 1 Wolfgang Kulhanek 2019-08-05 21:52:29 UTC

oc adm must-gather doesn't work:

the server is currently unable to handle the request (get imagestreams.image.openshift.io must-gather)
Using image: quay.io/openshift/origin-must-gather:latest
namespace/openshift-must-gather-k9rf2 created
clusterrolebinding.rbac.authorization.k8s.io/must-gather-bln94 created
clusterrolebinding.rbac.authorization.k8s.io/must-gather-bln94 deleted
namespace/openshift-must-gather-k9rf2 deleted
error: timed out waiting for the condition

Comment 4 David Eads 2019-08-29 20:20:54 UTC

the design is located in https://github.com/openshift/enhancements/pull/1 .  The short-term workaround is to 
1. `oc delete -n openshift-kube-controller-manager-operator secrets/csr-signer-signer secrets/csr-signer`,
2. then wait for 15 minutes. 
3. You should see several clusteroperator oc get co go to progressing pretty quick.  I think I saw it take two cycles for me locally, hence 15 minutes

Comment 5 Wolfgang Kulhanek 2019-08-29 22:34:52 UTC

Wrote a document how to set this up:
https://github.com/redhat-cop/openshift-lab-origin/blob/master/OpenShift4/Stopping_and_Resuming_OCP4_Clusters.adoc

Comment 7 Michal Fojtik 2019-11-07 17:43:32 UTC

Fix: https://github.com/openshift/cluster-kube-controller-manager-operator/pull/305

Comment 12 errata-xmlrpc 2020-01-23 11:05:01 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062