Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1836082

Summary: Installation fails on vSphere, AWS, and Azure IPI - secret "v4-0-config-system-router-certs" not found
Product: OpenShift Container Platform Reporter: jima
Component: NetworkingAssignee: Andrew McDermott <amcdermo>
Networking sub component: router QA Contact: Hongan Li <hongli>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: high CC: adahiya, anbhat, anusaxen, aos-bugs, gpei, juzhao, rbrattai, wsun, xtian, zzhao
Version: 4.5Keywords: TestBlocker
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-20 19:16:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
log-bundle-files none

Description jima 2020-05-15 06:02:39 UTC
Created attachment 1688776 [details]
log-bundle-files

Created attachment 1688776 [details]
log-bundle-files

Description of problem:
Install ocp4.5 on vpshere/ipi where enable ovn network, installation is failed.

In .openshift_install.log file:
time="2020-05-14T03:25:17-04:00" level=info msg="Waiting up to 40m0s for bootstrapping to complete..."
time="2020-05-14T04:05:18-04:00" level=error msg="Cluster operator authentication Degraded is True with ConfigObservation_Error::IngressStateEndpoints_MissingEndpoints::RouterCerts_NoRouterCertSecret: RouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server"
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator authentication Progressing is Unknown with NoData: "
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator authentication Available is Unknown with NoData: "
time="2020-05-14T04:05:18-04:00" level=error msg="Cluster operator kube-apiserver Degraded is True with StaticPods_Error: StaticPodsDegraded: pods \"kube-apiserver-jima-0514ovn-lhlhx-master-2\" not found\nStaticPodsDegraded: pods \"kube-apiserver-jima-0514ovn-lhlhx-master-0\" not found\nStaticPodsDegraded: pods \"kube-apiserver-jima-0514ovn-lhlhx-master-1\" not found"
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator kube-apiserver Progressing is True with NodeInstaller: NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2"
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator kube-apiserver Available is False with StaticPods_ZeroNodesActive: StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2"
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator kube-storage-version-migrator Available is False with _NoMigratorPod: Available: deployment/migrator.openshift-kube-storage-version-migrator: no replicas are available"
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator machine-config Progressing is True with : Unable to apply 4.5.0-0.nightly-2020-05-13-221558"
time="2020-05-14T04:05:18-04:00" level=error msg="Cluster operator machine-config Degraded is True with RenderConfigFailed: Unable to apply 4.5.0-0.nightly-2020-05-13-221558: openshift-config-managed/kube-cloud-config configmap is required on platform VSphere but not found: configmap \"kube-cloud-config\" not found"
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator machine-config Available is False with : Cluster not available for 4.5.0-0.nightly-2020-05-13-221558"
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator openshift-apiserver Available is False with APIServices_PreconditionNotReady: APIServicesAvailable: PreconditionNotReady"
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator openshift-controller-manager Progressing is True with _DesiredStateNotYetAchieved: Progressing: daemonset/controller-manager: observed generation is 0, desired generation is 8.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1"
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator openshift-controller-manager Available is False with _NoPodsAvailable: Available: no daemon pods available on any node."
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator operator-lifecycle-manager-packageserver Available is False with : "
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator operator-lifecycle-manager-packageserver Progressing is True with : Working toward 0.14.2"
time="2020-05-14T04:05:18-04:00" level=info msg="Cluster operator service-ca Progressing is True with _ManagedDeploymentsAvailable: Progressing: \nProgressing: service-ca does not have available replicas"

On bootstrap server, also found some errors:
May 14 07:46:10 dhcp-139-178-76-48 bootkube.sh[7967]: E0514 07:46:10.184146       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug=""
May 14 07:46:10 dhcp-139-178-76-48 bootkube.sh[7967]: E0514 07:46:10.187305       1 reflector.go:251] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to watch *v1.Pod: Get https://localhost:6443/api/v1/pods?watch=true: dial tcp [::1]:6443: connect: connection refused
May 14 08:05:17 dhcp-139-178-76-48 bootkube.sh[7967]: Error: error while checking pod status: timed out waiting for the condition
May 14 08:05:17 dhcp-139-178-76-48 bootkube.sh[7967]: Tearing down temporary bootstrap control plane...
May 14 08:05:17 dhcp-139-178-76-48 bootkube.sh[7967]: Error: error while checking pod status: timed out waiting for the condition


This issue is only happened when enabling ovn network on vsphere/ipi.

Version-Release number of selected component (if applicable):
4.5.0-0.nightly-2020-05-13-221558

How reproducible:
Always

Steps to Reproduce:
1. Install ipi ocp on vshpere and enabled ovn network
networking:
  networkType: OVNKubernetes
2.
3.

Actual results:
Installation is failed.

Expected results:
Installation is successful. 

Additional info:

Comment 1 Ross Brattain 2020-05-17 23:47:37 UTC
Also occurs on AWS and Azure.

seems to have started with release 4.5.0-0.nightly-2020-05-15-074717

also occurs with release
4.5.0-0.nightly-2020-05-15-105838
4.5.0-0.nightly-2020-05-17-171339

Comment 2 zhaozhanqi 2020-05-18 08:39:57 UTC
(In reply to Ross Brattain from comment #1)
> Also occurs on AWS and Azure.
> 
> seems to have started with release 4.5.0-0.nightly-2020-05-15-074717
> 
> also occurs with release
> 4.5.0-0.nightly-2020-05-15-105838
> 4.5.0-0.nightly-2020-05-17-171339

I checked this with build 4.5.0-0.nightly-2020-05-18-043243 on IPI AWS with OVN, it works well.

Comment 3 Ross Brattain 2020-05-18 12:11:56 UTC
Failed on IPI AWS OVN with 4.5.0-0.nightly-2020-05-18-072156

Comment 5 Anurag saxena 2020-05-18 15:38:36 UTC
Apparently a general issue while bringing up the clusters, irrespective of network plugin.

Comment 6 Abhinav Dahiya 2020-05-18 16:25:43 UTC
Moving to networking team to debug OVN,

Comment 7 Abhinav Dahiya 2020-05-18 16:27:09 UTC
Attach the oc adm must-gather logs

Comment 8 Abhinav Dahiya 2020-05-18 16:35:22 UTC
The failure is very similar to the error seen in https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-azure-compact-4.5/85

```
 level=error msg="Cluster operator authentication Degraded is True with ConfigObservation_Error::IngressStateEndpoints_MissingEndpoints::RouterCerts_NoRouterCertSecret: ConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server" 
```

```
 level=error msg="Cluster operator authentication Degraded is True with ConfigObservation_Error::IngressStateEndpoints_MissingEndpoints::RouterCerts_NoRouterCertSecret: ConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server" 
```

```
 level=error msg="Cluster operator machine-config Degraded is True with RenderConfigFailed: Unable to apply 4.5.0-0.ci-2020-05-18-100154: openshift-config-managed/kube-cloud-config configmap is required on platform Azure but not found: configmap \"kube-cloud-config\" not found" 
```

There is all kinds of broken in the log bundle https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-azure-compact-4.5/85/artifacts/e2e-azure/installer/log-bundle-20200518151239.tar

- the cluster version operator on bootstrap is failing to create a bunch of resources on API taking too long..