Version: $ openshift-install version openshift-install 4.8.0-0.nightly-2021-04-26-151924 built from commit fdf0269c32e313b6213ec36a84a5a849b43318f4 release image registry.ci.openshift.org/ocp/release@sha256:45a9d4b7e2a8e55ff20f1518436e965663161fab7cc3bec7e6c49376e2b62711 Platform: OpenStack Please specify: * IPI Installer fails to create workers in OpenStack: (shiftstack) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+-----------------------------+--------+-------------------------------------+--------------------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------------------+--------+-------------------------------------+--------------------+--------+ | 712c46bf-6d2c-4779-8a09-9637007e1ce9 | ostest-8cr65-worker-0-tgndg | ERROR | | ostest-8cr65-rhcos | | | f3033a66-e0d2-42d8-bd75-190f934ebd92 | ostest-8cr65-worker-0-9mbf8 | ERROR | | ostest-8cr65-rhcos | | | bcbbf313-177a-467d-90b7-8f264d0b2021 | ostest-8cr65-worker-0-6dpb9 | ERROR | | ostest-8cr65-rhcos | | | d8c1e447-d43c-43d6-bebe-dde8a8e12a9e | ostest-8cr65-master-2 | ACTIVE | ostest-8cr65-openshift=10.196.3.238 | ostest-8cr65-rhcos | | | 269ab17a-3216-4dd7-a966-7efa92ef74e5 | ostest-8cr65-master-1 | ACTIVE | ostest-8cr65-openshift=10.196.0.166 | ostest-8cr65-rhcos | | | 00d90ccf-8b92-4c03-865a-f51973371936 | ostest-8cr65-master-0 | ACTIVE | ostest-8cr65-openshift=10.196.3.229 | ostest-8cr65-rhcos | | +--------------------------------------+-----------------------------+--------+-------------------------------------+--------------------+--------+ time="2021-04-28T15:25:33Z" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.8.0-0.nightly-2021-04-26-151924: 647 of 677 done (95% complete)" time="2021-04-28T15:25:34Z" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.8.0-0.nightly-2021-04-26-151924: 649 of 677 done (95% complete)" time="2021-04-28T15:27:30Z" level=debug msg="Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, kube-apiserver, monitoring" time="2021-04-28T15:29:15Z" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.8.0-0.nightly-2021-04-26-151924: 654 of 677 done (96% complete)" time="2021-04-28T15:30:30Z" level=debug msg="Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring" time="2021-04-28T15:36:24Z" level=debug msg="Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring" time="2021-04-28T15:54:30Z" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.8.0-0.nightly-2021-04-26-151924: 655 of 677 done (96% complete)" time="2021-04-28T15:57:00Z" level=debug msg="Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring" time="2021-04-28T15:58:34Z" level=error msg="Cluster operator authentication Degraded is True with IngressStateEndpoints_MissingSubsets::OAuthClientsController_SyncError::OAuthServerDeployment_PreconditionNotFulfilled::OAuthServerRouteEndpointAccessibleController_SyncError::OAuthServerServiceEndpointAccessibleController_SyncError::OAuthServerServiceEndpointsEndpointAccessibleController_SyncError::Route_InvalidCanonicalHost::WellKnownReadyController_SyncError: IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthClientsControllerDegraded: no ingress for host oauth-openshift.apps.ostest.shiftstack.com in route oauth-openshift in namespace openshift-authentication\nOAuthServerDeploymentDegraded: waiting for the oauth-openshift route to contain an admitted ingress: no admitted ingress for route oauth-openshift in namespace openshift-authentication\nOAuthServerDeploymentDegraded: \nOAuthServerRouteEndpointAccessibleControllerDegraded: route \"openshift-authentication/oauth-openshift\": status does not have a host address\nOAuthServerServiceEndpointAccessibleControllerDegraded: Get \"https://172.30.178.123:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouteDegraded: no ingress for host oauth-openshift.apps.ostest.shiftstack.com in route oauth-openshift in namespace openshift-authentication\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" time="2021-04-28T15:58:34Z" level=info msg="Cluster operator authentication Available is False with OAuthServerDeployment_PreconditionNotFulfilled::OAuthServerServiceEndpointAccessibleController_EndpointUnavailable::OAuthServerServiceEndpointsEndpointAccessibleController_ResourceNotFound::ReadyIngressNodes_NoReadyIngressNodes::WellKnown_NotReady: OAuthServerServiceEndpointAccessibleControllerAvailable: Get \"https://172.30.178.123:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints \"oauth-openshift\" not found\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" time="2021-04-28T15:58:34Z" level=info msg="Cluster operator baremetal Disabled is True with UnsupportedPlatform: Nothing to do on this Platform" time="2021-04-28T15:58:34Z" level=info msg="Cluster operator console Progressing is True with DefaultRouteSync_FailedAdmitDefaultRoute::OAuthClientSync_FailedHost: DefaultRouteSyncProgressing: route \"console\" is not available at canonical host []\nOAuthClientSyncProgressing: route \"console\" is not available at canonical host []" time="2021-04-28T15:58:34Z" level=info msg="Cluster operator console Available is Unknown with NoData: " time="2021-04-28T15:58:34Z" level=info msg="Cluster operator image-registry Available is False with NoReplicasAvailable: Available: The deployment does not have available replicas\nNodeCADaemonAvailable: The daemon set node-ca has available replicas\nImagePrunerAvailable: Pruner CronJob has been created" time="2021-04-28T15:58:34Z" level=info msg="Cluster operator image-registry Progressing is True with DeploymentNotCompleted: Progressing: The deployment has not completed" time="2021-04-28T15:58:34Z" level=error msg="Cluster operator image-registry Degraded is True with Unavailable: Degraded: The deployment does not have available replicas" time="2021-04-28T15:58:34Z" level=info msg="Cluster operator ingress Available is False with IngressUnavailable: Not all ingress controllers are available." time="2021-04-28T15:58:34Z" level=info msg="Cluster operator ingress Progressing is True with Reconciling: Not all ingress controllers are available." time="2021-04-28T15:58:34Z" level=error msg="Cluster operator ingress Degraded is True with IngressControllersDegraded: Some ingresscontrollers are degraded: ingresscontroller \"default\" is degraded: DegradedConditions: One or more other status conditions indicate a degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled: Pod \"router-default-84fb65d5cd-xzgrg\" cannot be scheduled: 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate. Pod \"router-default-84fb65d5cd-c9vgw\" cannot be scheduled: 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate. Make sure you have sufficient worker nodes.), DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1)" time="2021-04-28T15:58:34Z" level=info msg="Cluster operator insights Disabled is False with AsExpected: " time="2021-04-28T15:58:34Z" level=error msg="Cluster operator monitoring Degraded is True with UpdatingAlertmanagerFailed: Failed to rollout the stack. Error: running task Updating Alertmanager failed: waiting for Alertmanager Route to become ready failed: waiting for route openshift-monitoring/alertmanager-main: no status available" time="2021-04-28T15:58:34Z" level=info msg="Cluster operator monitoring Available is False with UpdatingconfigurationsharingFailed: Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error." time="2021-04-28T15:58:34Z" level=info msg="Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack." time="2021-04-28T15:58:34Z" level=info msg="Cluster operator network ManagementStateDegraded is False with : " time="2021-04-28T15:58:34Z" level=info msg="Cluster operator network Progressing is True with Deploying: Deployment \"openshift-network-diagnostics/network-check-source\" is not available (awaiting 1 nodes)" time="2021-04-28T15:58:34Z" level=error msg="Cluster initialization failed because one or more operators are not functioning properly.\nThe cluster should be accessible for troubleshooting as detailed in the documentation linked below,\nhttps://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-installations.html\nThe 'wait-for install-complete' subcommand can then be used to continue the installation" time="2021-04-28T15:58:34Z" level=fatal msg="failed to initialize the cluster: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring" Must-gather too large to attach ClusterID: 32d672b8-cd10-4ce3-9b2e-308bfad98f86 ClusterVersion: Installing "4.8.0-0.nightly-2021-04-26-151924" for 4 days: Unable to apply 4.8.0-0.nightly-2021-04-26-151924: some cluster operators have not yet rolled out ClusterOperators: clusteroperator/authentication is not available (OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.178.123:443/healthz": dial tcp 172.30.178.123:443: i/o timeout (Client.Timeout exceeded while awaiting headers) OAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints "oauth-openshift" not found ReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods). WellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap "oauth-openshift" not found (check authentication operator, it is supposed to create this)) because IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server OAuthClientsControllerDegraded: no ingress for host oauth-openshift.apps.ostest.shiftstack.com in route oauth-openshift in namespace openshift-authentication OAuthServerDeploymentDegraded: waiting for the oauth-openshift route to contain an admitted ingress: no admitted ingress for route oauth-openshift in namespace openshift-authentication OAuthServerDeploymentDegraded: OAuthServerRouteEndpointAccessibleControllerDegraded: route "openshift-authentication/oauth-openshift": status does not have a host address OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.178.123:443/healthz": dial tcp 172.30.178.123:443: i/o timeout (Client.Timeout exceeded while awaiting headers) OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready RouteDegraded: no ingress for host oauth-openshift.apps.ostest.shiftstack.com in route oauth-openshift in namespace openshift-authentication WellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap "oauth-openshift" not found (check authentication operator, it is supposed to create this) clusteroperator/console is not available () because All is well clusteroperator/image-registry is not available (Available: The deployment does not have available replicas NodeCADaemonAvailable: The daemon set node-ca has available replicas ImagePrunerAvailable: Pruner CronJob has been created) because Degraded: The deployment does not have available replicas clusteroperator/ingress is not available (Not all ingress controllers are available.) because Some ingresscontrollers are degraded: ingresscontroller "default" is degraded: DegradedConditions: One or more other status conditions indicate a degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled: Pod "router-default-84fb65d5cd-xzgrg" cannot be scheduled: 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate. Pod "router-default-84fb65d5cd-c9vgw" cannot be scheduled: 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate. Make sure you have sufficient worker nodes.), DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1), DeploymentReplicasAllAvailable=False (DeploymentReplicasNotAvailable: 0/2 of replicas are available) clusteroperator/monitoring is not available (Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.) because Failed to rollout the stack. Error: running task Updating Alertmanager failed: waiting for Alertmanager Route to become ready failed: waiting for route openshift-monitoring/alertmanager-main: no status available clusteroperator/network is progressing: Deployment "openshift-network-diagnostics/network-check-source" is not available (awaiting 1 nodes) What did you expect to happen? Installation should pass How to reproduce it (as minimally and precisely as possible)? 100% Anything else we need to know? install-config: # This file is autogenerated by infrared openshift plugin apiVersion: v1 baseDomain: "shiftstack.com" clusterID: "2539313a-8f78-5205-919e-ed0d8ae76763" compute: - name: worker platform: openstack: zones: [] additionalNetworkIDs: ['1ae8b499-30de-4122-b355-d6562dbfd0a9'] replicas: 3 controlPlane: name: master platform: openstack: zones: [] replicas: 3 metadata: name: "ostest" networking: clusterNetworks: - cidr: 10.128.0.0/14 hostSubnetLength: 9 serviceCIDR: 172.30.0.0/16 machineCIDR: 10.196.0.0/16 type: "Kuryr" platform: openstack: cloud: "shiftstack" externalNetwork: "nova" region: "regionOne" computeFlavor: "m4.xlarge" lbFloatingIP: "10.46.22.183" ingressFloatingIP: "10.46.22.178" externalDNS: ["10.46.0.31"] pullSecret: |
I see that your worker nodes all failed to come up. When you view those nodes in openstack, what reason does it give for them failing? The installer is failing because it cant deploy the OpenShift services without workers, unless you make the masters schedulable.
So I think this is related to https://bugzilla.redhat.com/show_bug.cgi?id=1955969
ah yes, there seems to be a regression in that right now. To unblock yourself, try to work around the issue by using the ports directive instead. https://github.com/openshift/cluster-api-provider-openstack/blob/f17f967e01975ac0142f4d8a8011c9caca8b61d6/pkg/apis/openstackproviderconfig/v1alpha1/types.go#L201-L234
Closing as duplicate. https://bugzilla.redhat.com/show_bug.cgi?id=1955969 is also marked as a test blocker. *** This bug has been marked as a duplicate of bug 1955969 ***