Bug 1850099

Summary:	OCP 4.4.8 failed to generate asset "Platform Permissions Check"
Product:	OpenShift Container Platform	Reporter:	Noam Manos <nmanos>
Component:	Installer	Assignee:	Abhinav Dahiya <adahiya>
Installer sub component:	openshift-installer	QA Contact:	Johnny Liu <jialiu>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	bleanhar, oblaut, tfreger, wking
Version:	4.4
Target Milestone:	---
Target Release:	4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-06-24 12:17:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Noam Manos 2020-06-23 14:33:59 UTC

Description of problem:
While installing Openshift 4.4.8 on AWS (on us-east-1 region), I got fatal error, and installation has failed:

failed to fetch Cluster:
failed to fetch dependency of \"Cluster\":
failed to generate asset \"Platform Permissions Check\":
validate AWS credentials: mint credentials check:
error simulating policy:
Throttling: Rate exceeded
status code: 400, request id: e19d7825-d6a3-4a69-afb2-87ed38cf040e


Version-Release number of selected component (if applicable):
OCP 4.4.8

How reproducible:
Few times already

Steps to Reproduce:
Download and run OCP installer 4.4.8

Actual results:

$ ./openshift-install create cluster --log-level debug

level=debug msg="OpenShift Installer 4.4.8"
level=debug msg="Built from commit 1541bf917973186bbab6a5f895f08db4334a5d9a"
level=debug msg="Fetching Terraform Variables..."
level=debug msg="Loading Terraform Variables..."
level=debug msg="  Loading Cluster ID..."
level=debug msg="    Loading Install Config..."
level=debug msg="      Loading SSH Key..."
level=debug msg="      Loading Base Domain..."
level=debug msg="        Loading Platform..."
level=debug msg="      Loading Cluster Name..."
level=debug msg="        Loading Base Domain..."
level=debug msg="        Loading Platform..."
level=debug msg="      Loading Pull Secret..."
level=debug msg="      Loading Platform..."
level=debug msg="    Using Install Config loaded from target directory"
level=debug msg="  Loading Install Config..."
level=debug msg="  Loading Image..."
level=debug msg="    Loading Install Config..."
level=debug msg="  Loading BootstrapImage..."
level=debug msg="    Loading Install Config..."
level=debug msg="  Loading Bootstrap Ignition Config..."
level=debug msg="    Loading Install Config..."
level=debug msg="    Loading Kubeconfig Admin Internal Client..."
level=debug msg="      Loading Certificate (admin-kubeconfig-client)..."
level=debug msg="        Loading Certificate (admin-kubeconfig-signer)..."
level=debug msg="      Loading Certificate (kube-apiserver-complete-server-ca-bundle)..."
level=debug msg="        Loading Certificate (kube-apiserver-localhost-ca-bundle)..."
level=debug msg="          Loading Certificate (kube-apiserver-localhost-signer)..."
level=debug msg="        Loading Certificate (kube-apiserver-service-network-ca-bundle)..."
level=debug msg="          Loading Certificate (kube-apiserver-service-network-signer)..."
level=debug msg="        Loading Certificate (kube-apiserver-lb-ca-bundle)..."
level=debug msg="          Loading Certificate (kube-apiserver-lb-signer)..."
level=debug msg="      Loading Install Config..."
level=debug msg="    Loading Kubeconfig Kubelet..."
level=debug msg="      Loading Certificate (kube-apiserver-complete-server-ca-bundle)..."
level=debug msg="      Loading Certificate (kubelet-client)..."
level=debug msg="        Loading Certificate (kubelet-bootstrap-kubeconfig-signer)..."
level=debug msg="      Loading Install Config..."
level=debug msg="    Loading Kubeconfig Admin Client (Loopback)..."
level=debug msg="      Loading Certificate (admin-kubeconfig-client)..."
level=debug msg="      Loading Certificate (kube-apiserver-localhost-ca-bundle)..."
level=debug msg="      Loading Install Config..."
level=debug msg="    Loading Master Machines..."
level=debug msg="      Loading Cluster ID..."
level=debug msg="      Loading Platform Credentials Check..."
level=debug msg="        Loading Install Config..."
level=debug msg="      Loading Install Config..."
level=debug msg="      Loading Image..."
level=debug msg="      Loading Master Ignition Config..."
level=debug msg="        Loading Install Config..."
level=debug msg="        Loading Root CA..."
level=debug msg="    Loading Worker Machines..."
level=debug msg="      Loading Cluster ID..."
level=debug msg="      Loading Platform Credentials Check..."
level=debug msg="      Loading Install Config..."
level=debug msg="      Loading Image..."
level=debug msg="      Loading Worker Ignition Config..."
level=debug msg="        Loading Install Config..."
level=debug msg="        Loading Root CA..."
level=debug msg="    Loading Common Manifests..."
level=debug msg="      Loading Cluster ID..."
level=debug msg="      Loading Install Config..."
level=debug msg="      Loading Ingress Config..."
level=debug msg="        Loading Install Config..."
level=debug msg="      Loading DNS Config..."
level=debug msg="        Loading Install Config..."
level=debug msg="        Loading Cluster ID..."
level=debug msg="        Loading Platform Credentials Check..."
level=debug msg="      Loading Infrastructure Config..."
level=debug msg="        Loading Cluster ID..."
level=debug msg="        Loading Install Config..."
level=debug msg="        Loading Cloud Provider Config..."
level=debug msg="          Loading Install Config..."
level=debug msg="          Loading Cluster ID..."
level=debug msg="          Loading Platform Credentials Check..."
level=debug msg="        Loading Additional Trust Bundle Config..."
level=debug msg="          Loading Install Config..."
level=debug msg="      Loading Network Config..."
level=debug msg="        Loading Install Config..."
level=debug msg="        Loading Network CRDs..."
level=debug msg="      Loading Proxy Config..."
level=debug msg="        Loading Install Config..."
level=debug msg="        Loading Network Config..."
level=debug msg="      Loading Scheduler Config..."
level=debug msg="        Loading Install Config..."
level=debug msg="      Loading Image Content Source Policy..."
level=debug msg="        Loading Install Config..."
level=debug msg="      Loading Root CA..."
level=debug msg="      Loading Certificate (etcd-signer)..."
level=debug msg="      Loading Certificate (etcd-ca-bundle)..."
level=debug msg="        Loading Certificate (etcd-signer)..."
level=debug msg="      Loading Certificate (etcd-client)..."
level=debug msg="        Loading Certificate (etcd-signer)..."
level=debug msg="      Loading Certificate (etcd-metric-ca-bundle)..."
level=debug msg="        Loading Certificate (etcd-metric-signer)..."
level=debug msg="      Loading Certificate (etcd-metric-signer)..."
level=debug msg="      Loading Certificate (etcd-metric-signer-client)..."
level=debug msg="        Loading Certificate (etcd-metric-signer)..."
level=debug msg="      Loading Certificate (mcs)..."
level=debug msg="        Loading Root CA..."
level=debug msg="        Loading Install Config..."
level=debug msg="      Loading CVOOverrides..."
level=debug msg="      Loading EtcdCAConfigMap..."
level=debug msg="      Loading EtcdClientSecret..."
level=debug msg="      Loading EtcdHostServiceEndpoints..."
level=debug msg="      Loading EtcdHostService..."
level=debug msg="      Loading EtcdMetricClientSecret..."
level=debug msg="      Loading EtcdMetricServingCAConfigMap..."
level=debug msg="      Loading EtcdMetricSignerSecret..."
level=debug msg="      Loading EtcdNamespace..."
level=debug msg="      Loading EtcdService..."
level=debug msg="      Loading EtcdSignerSecret..."
level=debug msg="      Loading KubeCloudConfig..."
level=debug msg="      Loading EtcdServingCAConfigMap..."
level=debug msg="      Loading KubeSystemConfigmapRootCA..."
level=debug msg="      Loading MachineConfigServerTLSSecret..."
level=debug msg="      Loading OpenshiftConfigSecretPullSecret..."
level=debug msg="      Loading OpenshiftMachineConfigOperator..."
level=debug msg="    Loading Openshift Manifests..."
level=debug msg="      Loading Install Config..."
level=debug msg="      Loading Cluster ID..."
level=debug msg="      Loading Kubeadmin Password..."
level=debug msg="      Loading OpenShift Install (Manifests)..."
level=debug msg="      Loading CloudCredsSecret..."
level=debug msg="      Loading KubeadminPasswordSecret..."
level=debug msg="      Loading RoleCloudCredsSecretReader..."
level=debug msg="      Loading Private Cluster Outbound Service..."
level=debug msg="      Loading Baremetal Config CR..."
level=debug msg="      Loading Image..."
level=debug msg="    Loading Proxy Config..."
level=debug msg="    Loading Certificate (admin-kubeconfig-ca-bundle)..."
level=debug msg="      Loading Certificate (admin-kubeconfig-signer)..."
level=debug msg="    Loading Certificate (aggregator)..."
level=debug msg="    Loading Certificate (aggregator-ca-bundle)..."
level=debug msg="      Loading Certificate (aggregator-signer)..."
level=debug msg="    Loading Certificate (system:kube-apiserver-proxy)..."
level=debug msg="      Loading Certificate (aggregator-signer)..."
level=debug msg="    Loading Certificate (aggregator-signer)..."
level=debug msg="    Loading Certificate (system:kube-apiserver-proxy)..."
level=debug msg="      Loading Certificate (aggregator)..."
level=debug msg="    Loading Certificate (etcd-ca-bundle)..."
level=debug msg="    Loading Certificate (etcd-metric-ca-bundle)..."
level=debug msg="    Loading Certificate (etcd-metric-signer)..."
level=debug msg="    Loading Certificate (etcd-metric-signer-client)..."
level=debug msg="    Loading Certificate (etcd-signer)..."
level=debug msg="    Loading Certificate (etcd-client)..."
level=debug msg="    Loading Certificate (journal-gatewayd)..."
level=debug msg="      Loading Root CA..."
level=debug msg="    Loading Certificate (kube-apiserver-lb-ca-bundle)..."
level=debug msg="    Loading Certificate (kube-apiserver-external-lb-server)..."
level=debug msg="      Loading Certificate (kube-apiserver-lb-signer)..."
level=debug msg="      Loading Install Config..."
level=debug msg="    Loading Certificate (kube-apiserver-internal-lb-server)..."
level=debug msg="      Loading Certificate (kube-apiserver-lb-signer)..."
level=debug msg="      Loading Install Config..."
level=debug msg="    Loading Certificate (kube-apiserver-lb-signer)..."
level=debug msg="    Loading Certificate (kube-apiserver-localhost-ca-bundle)..."
level=debug msg="    Loading Certificate (kube-apiserver-localhost-server)..."
level=debug msg="      Loading Certificate (kube-apiserver-localhost-signer)..."
level=debug msg="    Loading Certificate (kube-apiserver-localhost-signer)..."
level=debug msg="    Loading Certificate (kube-apiserver-service-network-ca-bundle)..."
level=debug msg="    Loading Certificate (kube-apiserver-service-network-server)..."
level=debug msg="      Loading Certificate (kube-apiserver-service-network-signer)..."
level=debug msg="      Loading Install Config..."
level=debug msg="    Loading Certificate (kube-apiserver-service-network-signer)..."
level=debug msg="    Loading Certificate (kube-apiserver-complete-server-ca-bundle)..."
level=debug msg="    Loading Certificate (kube-apiserver-complete-client-ca-bundle)..."
level=debug msg="      Loading Certificate (admin-kubeconfig-ca-bundle)..."
level=debug msg="      Loading Certificate (kubelet-client-ca-bundle)..."
level=debug msg="        Loading Certificate (kubelet-signer)..."
level=debug msg="      Loading Certificate (kube-control-plane-ca-bundle)..."
level=debug msg="        Loading Certificate (kube-control-plane-signer)..."
level=debug msg="        Loading Certificate (kube-apiserver-lb-signer)..."
level=debug msg="        Loading Certificate (kube-apiserver-localhost-signer)..."
level=debug msg="        Loading Certificate (kube-apiserver-service-network-signer)..."
level=debug msg="      Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)..."
level=debug msg="        Loading Certificate (kube-apiserver-to-kubelet-signer)..."
level=debug msg="      Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)..."
level=debug msg="        Loading Certificate (kubelet-bootstrap-kubeconfig-signer)..."
level=debug msg="    Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)..."
level=debug msg="    Loading Certificate (kube-apiserver-to-kubelet-client)..."
level=debug msg="      Loading Certificate (kube-apiserver-to-kubelet-signer)..."
level=debug msg="    Loading Certificate (kube-apiserver-to-kubelet-signer)..."
level=debug msg="    Loading Certificate (kube-control-plane-ca-bundle)..."
level=debug msg="    Loading Certificate (kube-control-plane-kube-controller-manager-client)..."
level=debug msg="      Loading Certificate (kube-control-plane-signer)..."
level=debug msg="    Loading Certificate (kube-control-plane-kube-scheduler-client)..."
level=debug msg="      Loading Certificate (kube-control-plane-signer)..."
level=debug msg="    Loading Certificate (kube-control-plane-signer)..."
level=debug msg="    Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)..."
level=debug msg="    Loading Certificate (kubelet-client-ca-bundle)..."
level=debug msg="    Loading Certificate (kubelet-client)..."
level=debug msg="    Loading Certificate (kubelet-signer)..."
level=debug msg="    Loading Certificate (kubelet-serving-ca-bundle)..."
level=debug msg="      Loading Certificate (kubelet-signer)..."
level=debug msg="    Loading Certificate (mcs)..."
level=debug msg="    Loading Root CA..."
level=debug msg="    Loading Key Pair (service-account.pub)..."
level=debug msg="    Loading Release Image Pull Spec..."
level=debug msg="    Loading Image..."
level=debug msg="  Loading Master Ignition Config..."
level=debug msg="  Loading Master Machines..."
level=debug msg="  Loading Worker Machines..."
level=debug msg="  Fetching Cluster ID..."
level=debug msg="    Fetching Install Config..."
level=debug msg="    Reusing previously-fetched Install Config"
level=debug msg="  Generating Cluster ID..."
level=debug msg="  Fetching Install Config..."
level=debug msg="  Reusing previously-fetched Install Config"
level=debug msg="  Fetching Image..."
level=debug msg="    Fetching Install Config..."
level=debug msg="    Reusing previously-fetched Install Config"
level=debug msg="  Generating Image..."
level=debug msg="  Fetching BootstrapImage..."
level=debug msg="    Fetching Install Config..."
level=debug msg="    Reusing previously-fetched Install Config"
level=debug msg="  Generating BootstrapImage..."
level=debug msg="  Fetching Bootstrap Ignition Config..."
level=debug msg="    Fetching Install Config..."
level=debug msg="    Reusing previously-fetched Install Config"
level=debug msg="    Fetching Kubeconfig Admin Internal Client..."
level=debug msg="      Fetching Certificate (admin-kubeconfig-client)..."
level=debug msg="        Fetching Certificate (admin-kubeconfig-signer)..."
level=debug msg="        Generating Certificate (admin-kubeconfig-signer)..."
level=debug msg="      Generating Certificate (admin-kubeconfig-client)..."
level=debug msg="      Fetching Certificate (kube-apiserver-complete-server-ca-bundle)..."
level=debug msg="        Fetching Certificate (kube-apiserver-localhost-ca-bundle)..."
level=debug msg="          Fetching Certificate (kube-apiserver-localhost-signer)..."
level=debug msg="          Generating Certificate (kube-apiserver-localhost-signer)..."
level=debug msg="        Generating Certificate (kube-apiserver-localhost-ca-bundle)..."
level=debug msg="        Fetching Certificate (kube-apiserver-service-network-ca-bundle)..."
level=debug msg="          Fetching Certificate (kube-apiserver-service-network-signer)..."
level=debug msg="          Generating Certificate (kube-apiserver-service-network-signer)..."
level=debug msg="        Generating Certificate (kube-apiserver-service-network-ca-bundle)..."
level=debug msg="        Fetching Certificate (kube-apiserver-lb-ca-bundle)..."
level=debug msg="          Fetching Certificate (kube-apiserver-lb-signer)..."
level=debug msg="          Generating Certificate (kube-apiserver-lb-signer)..."
level=debug msg="        Generating Certificate (kube-apiserver-lb-ca-bundle)..."
level=debug msg="      Generating Certificate (kube-apiserver-complete-server-ca-bundle)..."
level=debug msg="      Fetching Install Config..."
level=debug msg="      Reusing previously-fetched Install Config"
level=debug msg="    Generating Kubeconfig Admin Internal Client..."
level=debug msg="    Fetching Kubeconfig Kubelet..."
level=debug msg="      Fetching Certificate (kube-apiserver-complete-server-ca-bundle)..."
level=debug msg="      Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle)"
level=debug msg="      Fetching Certificate (kubelet-client)..."
level=debug msg="        Fetching Certificate (kubelet-bootstrap-kubeconfig-signer)..."
level=debug msg="        Generating Certificate (kubelet-bootstrap-kubeconfig-signer)..."
level=debug msg="      Generating Certificate (kubelet-client)..."
level=debug msg="      Fetching Install Config..."
level=debug msg="      Reusing previously-fetched Install Config"
level=debug msg="    Generating Kubeconfig Kubelet..."
level=debug msg="    Fetching Kubeconfig Admin Client (Loopback)..."
level=debug msg="      Fetching Certificate (admin-kubeconfig-client)..."
level=debug msg="      Reusing previously-fetched Certificate (admin-kubeconfig-client)"
level=debug msg="      Fetching Certificate (kube-apiserver-localhost-ca-bundle)..."
level=debug msg="      Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle)"
level=debug msg="      Fetching Install Config..."
level=debug msg="      Reusing previously-fetched Install Config"
level=debug msg="    Generating Kubeconfig Admin Client (Loopback)..."
level=debug msg="    Fetching Master Machines..."
level=debug msg="      Fetching Cluster ID..."
level=debug msg="      Reusing previously-fetched Cluster ID"
level=debug msg="      Fetching Platform Credentials Check..."
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="      Generating Platform Credentials Check..."
level=info msg="Credentials loaded from the \"default\" profile in file \"/home/jenkins/.aws/credentials\""
level=debug msg="      Fetching Install Config..."
level=debug msg="      Reusing previously-fetched Install Config"
level=debug msg="      Fetching Image..."
level=debug msg="      Reusing previously-fetched Image"
level=debug msg="      Fetching Master Ignition Config..."
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="        Fetching Root CA..."
level=debug msg="        Generating Root CA..."
level=debug msg="      Generating Master Ignition Config..."
level=debug msg="    Generating Master Machines..."
level=debug msg="    Fetching Worker Machines..."
level=debug msg="      Fetching Cluster ID..."
level=debug msg="      Reusing previously-fetched Cluster ID"
level=debug msg="      Fetching Platform Credentials Check..."
level=debug msg="      Reusing previously-fetched Platform Credentials Check"
level=debug msg="      Fetching Install Config..."
level=debug msg="      Reusing previously-fetched Install Config"
level=debug msg="      Fetching Image..."
level=debug msg="      Reusing previously-fetched Image"
level=debug msg="      Fetching Worker Ignition Config..."
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="        Fetching Root CA..."
level=debug msg="        Reusing previously-fetched Root CA"
level=debug msg="      Generating Worker Ignition Config..."
level=debug msg="    Generating Worker Machines..."
level=debug msg="    Fetching Common Manifests..."
level=debug msg="      Fetching Cluster ID..."
level=debug msg="      Reusing previously-fetched Cluster ID"
level=debug msg="      Fetching Install Config..."
level=debug msg="      Reusing previously-fetched Install Config"
level=debug msg="      Fetching Ingress Config..."
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="      Generating Ingress Config..."
level=debug msg="      Fetching DNS Config..."
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="        Fetching Cluster ID..."
level=debug msg="        Reusing previously-fetched Cluster ID"
level=debug msg="        Fetching Platform Credentials Check..."
level=debug msg="        Reusing previously-fetched Platform Credentials Check"
level=debug msg="      Generating DNS Config..."
level=debug msg="      Fetching Infrastructure Config..."
level=debug msg="        Fetching Cluster ID..."
level=debug msg="        Reusing previously-fetched Cluster ID"
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="        Fetching Cloud Provider Config..."
level=debug msg="          Fetching Install Config..."
level=debug msg="          Reusing previously-fetched Install Config"
level=debug msg="          Fetching Cluster ID..."
level=debug msg="          Reusing previously-fetched Cluster ID"
level=debug msg="          Fetching Platform Credentials Check..."
level=debug msg="          Reusing previously-fetched Platform Credentials Check"
level=debug msg="        Generating Cloud Provider Config..."
level=debug msg="        Fetching Additional Trust Bundle Config..."
level=debug msg="          Fetching Install Config..."
level=debug msg="          Reusing previously-fetched Install Config"
level=debug msg="        Generating Additional Trust Bundle Config..."
level=debug msg="      Generating Infrastructure Config..."
level=debug msg="      Fetching Network Config..."
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="        Fetching Network CRDs..."
level=debug msg="        Generating Network CRDs..."
level=debug msg="      Generating Network Config..."
level=debug msg="      Fetching Proxy Config..."
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="        Fetching Network Config..."
level=debug msg="        Reusing previously-fetched Network Config"
level=debug msg="      Generating Proxy Config..."
level=debug msg="      Fetching Scheduler Config..."
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="      Generating Scheduler Config..."
level=debug msg="      Fetching Image Content Source Policy..."
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="      Generating Image Content Source Policy..."
level=debug msg="      Fetching Root CA..."
level=debug msg="      Reusing previously-fetched Root CA"
level=debug msg="      Fetching Certificate (etcd-signer)..."
level=debug msg="      Generating Certificate (etcd-signer)..."
level=debug msg="      Fetching Certificate (etcd-ca-bundle)..."
level=debug msg="        Fetching Certificate (etcd-signer)..."
level=debug msg="        Reusing previously-fetched Certificate (etcd-signer)"
level=debug msg="      Generating Certificate (etcd-ca-bundle)..."
level=debug msg="      Fetching Certificate (etcd-client)..."
level=debug msg="        Fetching Certificate (etcd-signer)..."
level=debug msg="        Reusing previously-fetched Certificate (etcd-signer)"
level=debug msg="      Generating Certificate (etcd-client)..."
level=debug msg="      Fetching Certificate (etcd-metric-ca-bundle)..."
level=debug msg="        Fetching Certificate (etcd-metric-signer)..."
level=debug msg="        Generating Certificate (etcd-metric-signer)..."
level=debug msg="      Generating Certificate (etcd-metric-ca-bundle)..."
level=debug msg="      Fetching Certificate (etcd-metric-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (etcd-metric-signer)"
level=debug msg="      Fetching Certificate (etcd-metric-signer-client)..."
level=debug msg="        Fetching Certificate (etcd-metric-signer)..."
level=debug msg="        Reusing previously-fetched Certificate (etcd-metric-signer)"
level=debug msg="      Generating Certificate (etcd-metric-signer-client)..."
level=debug msg="      Fetching Certificate (mcs)..."
level=debug msg="        Fetching Root CA..."
level=debug msg="        Reusing previously-fetched Root CA"
level=debug msg="        Fetching Install Config..."
level=debug msg="        Reusing previously-fetched Install Config"
level=debug msg="      Generating Certificate (mcs)..."
level=debug msg="      Fetching CVOOverrides..."
level=debug msg="      Generating CVOOverrides..."
level=debug msg="      Fetching EtcdCAConfigMap..."
level=debug msg="      Generating EtcdCAConfigMap..."
level=debug msg="      Fetching EtcdClientSecret..."
level=debug msg="      Generating EtcdClientSecret..."
level=debug msg="      Fetching EtcdHostServiceEndpoints..."
level=debug msg="      Generating EtcdHostServiceEndpoints..."
level=debug msg="      Fetching EtcdHostService..."
level=debug msg="      Generating EtcdHostService..."
level=debug msg="      Fetching EtcdMetricClientSecret..."
level=debug msg="      Generating EtcdMetricClientSecret..."
level=debug msg="      Fetching EtcdMetricServingCAConfigMap..."
level=debug msg="      Generating EtcdMetricServingCAConfigMap..."
level=debug msg="      Fetching EtcdMetricSignerSecret..."
level=debug msg="      Generating EtcdMetricSignerSecret..."
level=debug msg="      Fetching EtcdNamespace..."
level=debug msg="      Generating EtcdNamespace..."
level=debug msg="      Fetching EtcdService..."
level=debug msg="      Generating EtcdService..."
level=debug msg="      Fetching EtcdSignerSecret..."
level=debug msg="      Generating EtcdSignerSecret..."
level=debug msg="      Fetching KubeCloudConfig..."
level=debug msg="      Generating KubeCloudConfig..."
level=debug msg="      Fetching EtcdServingCAConfigMap..."
level=debug msg="      Generating EtcdServingCAConfigMap..."
level=debug msg="      Fetching KubeSystemConfigmapRootCA..."
level=debug msg="      Generating KubeSystemConfigmapRootCA..."
level=debug msg="      Fetching MachineConfigServerTLSSecret..."
level=debug msg="      Generating MachineConfigServerTLSSecret..."
level=debug msg="      Fetching OpenshiftConfigSecretPullSecret..."
level=debug msg="      Generating OpenshiftConfigSecretPullSecret..."
level=debug msg="      Fetching OpenshiftMachineConfigOperator..."
level=debug msg="      Generating OpenshiftMachineConfigOperator..."
level=debug msg="    Generating Common Manifests..."
level=debug msg="    Fetching Openshift Manifests..."
level=debug msg="      Fetching Install Config..."
level=debug msg="      Reusing previously-fetched Install Config"
level=debug msg="      Fetching Cluster ID..."
level=debug msg="      Reusing previously-fetched Cluster ID"
level=debug msg="      Fetching Kubeadmin Password..."
level=debug msg="      Generating Kubeadmin Password..."
level=debug msg="      Fetching OpenShift Install (Manifests)..."
level=debug msg="      Generating OpenShift Install (Manifests)..."
level=debug msg="      Fetching CloudCredsSecret..."
level=debug msg="      Generating CloudCredsSecret..."
level=debug msg="      Fetching KubeadminPasswordSecret..."
level=debug msg="      Generating KubeadminPasswordSecret..."
level=debug msg="      Fetching RoleCloudCredsSecretReader..."
level=debug msg="      Generating RoleCloudCredsSecretReader..."
level=debug msg="      Fetching Private Cluster Outbound Service..."
level=debug msg="      Generating Private Cluster Outbound Service..."
level=debug msg="      Fetching Baremetal Config CR..."
level=debug msg="      Generating Baremetal Config CR..."
level=debug msg="      Fetching Image..."
level=debug msg="      Reusing previously-fetched Image"
level=debug msg="    Generating Openshift Manifests..."
level=debug msg="    Fetching Proxy Config..."
level=debug msg="    Reusing previously-fetched Proxy Config"
level=debug msg="    Fetching Certificate (admin-kubeconfig-ca-bundle)..."
level=debug msg="      Fetching Certificate (admin-kubeconfig-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (admin-kubeconfig-signer)"
level=debug msg="    Generating Certificate (admin-kubeconfig-ca-bundle)..."
level=debug msg="    Fetching Certificate (aggregator)..."
level=debug msg="    Generating Certificate (aggregator)..."
level=debug msg="    Fetching Certificate (aggregator-ca-bundle)..."
level=debug msg="      Fetching Certificate (aggregator-signer)..."
level=debug msg="      Generating Certificate (aggregator-signer)..."
level=debug msg="    Generating Certificate (aggregator-ca-bundle)..."
level=debug msg="    Fetching Certificate (system:kube-apiserver-proxy)..."
level=debug msg="      Fetching Certificate (aggregator-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (aggregator-signer)"
level=debug msg="    Generating Certificate (system:kube-apiserver-proxy)..."
level=debug msg="    Fetching Certificate (aggregator-signer)..."
level=debug msg="    Reusing previously-fetched Certificate (aggregator-signer)"
level=debug msg="    Fetching Certificate (system:kube-apiserver-proxy)..."
level=debug msg="      Fetching Certificate (aggregator)..."
level=debug msg="      Reusing previously-fetched Certificate (aggregator)"
level=debug msg="    Generating Certificate (system:kube-apiserver-proxy)..."
level=debug msg="    Fetching Certificate (etcd-ca-bundle)..."
level=debug msg="    Reusing previously-fetched Certificate (etcd-ca-bundle)"
level=debug msg="    Fetching Certificate (etcd-metric-ca-bundle)..."
level=debug msg="    Reusing previously-fetched Certificate (etcd-metric-ca-bundle)"
level=debug msg="    Fetching Certificate (etcd-metric-signer)..."
level=debug msg="    Reusing previously-fetched Certificate (etcd-metric-signer)"
level=debug msg="    Fetching Certificate (etcd-metric-signer-client)..."
level=debug msg="    Reusing previously-fetched Certificate (etcd-metric-signer-client)"
level=debug msg="    Fetching Certificate (etcd-signer)..."
level=debug msg="    Reusing previously-fetched Certificate (etcd-signer)"
level=debug msg="    Fetching Certificate (etcd-client)..."
level=debug msg="    Reusing previously-fetched Certificate (etcd-client)"
level=debug msg="    Fetching Certificate (journal-gatewayd)..."
level=debug msg="      Fetching Root CA..."
level=debug msg="      Reusing previously-fetched Root CA"
level=debug msg="    Generating Certificate (journal-gatewayd)..."
level=debug msg="    Fetching Certificate (kube-apiserver-lb-ca-bundle)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-apiserver-lb-ca-bundle)"
level=debug msg="    Fetching Certificate (kube-apiserver-external-lb-server)..."
level=debug msg="      Fetching Certificate (kube-apiserver-lb-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (kube-apiserver-lb-signer)"
level=debug msg="      Fetching Install Config..."
level=debug msg="      Reusing previously-fetched Install Config"
level=debug msg="    Generating Certificate (kube-apiserver-external-lb-server)..."
level=debug msg="    Fetching Certificate (kube-apiserver-internal-lb-server)..."
level=debug msg="      Fetching Certificate (kube-apiserver-lb-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (kube-apiserver-lb-signer)"
level=debug msg="      Fetching Install Config..."
level=debug msg="      Reusing previously-fetched Install Config"
level=debug msg="    Generating Certificate (kube-apiserver-internal-lb-server)..."
level=debug msg="    Fetching Certificate (kube-apiserver-lb-signer)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-apiserver-lb-signer)"
level=debug msg="    Fetching Certificate (kube-apiserver-localhost-ca-bundle)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle)"
level=debug msg="    Fetching Certificate (kube-apiserver-localhost-server)..."
level=debug msg="      Fetching Certificate (kube-apiserver-localhost-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (kube-apiserver-localhost-signer)"
level=debug msg="    Generating Certificate (kube-apiserver-localhost-server)..."
level=debug msg="    Fetching Certificate (kube-apiserver-localhost-signer)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-apiserver-localhost-signer)"
level=debug msg="    Fetching Certificate (kube-apiserver-service-network-ca-bundle)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-apiserver-service-network-ca-bundle)"
level=debug msg="    Fetching Certificate (kube-apiserver-service-network-server)..."
level=debug msg="      Fetching Certificate (kube-apiserver-service-network-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (kube-apiserver-service-network-signer)"
level=debug msg="      Fetching Install Config..."
level=debug msg="      Reusing previously-fetched Install Config"
level=debug msg="    Generating Certificate (kube-apiserver-service-network-server)..."
level=debug msg="    Fetching Certificate (kube-apiserver-service-network-signer)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-apiserver-service-network-signer)"
level=debug msg="    Fetching Certificate (kube-apiserver-complete-server-ca-bundle)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle)"
level=debug msg="    Fetching Certificate (kube-apiserver-complete-client-ca-bundle)..."
level=debug msg="      Fetching Certificate (admin-kubeconfig-ca-bundle)..."
level=debug msg="      Reusing previously-fetched Certificate (admin-kubeconfig-ca-bundle)"
level=debug msg="      Fetching Certificate (kubelet-client-ca-bundle)..."
level=debug msg="        Fetching Certificate (kubelet-signer)..."
level=debug msg="        Generating Certificate (kubelet-signer)..."
level=debug msg="      Generating Certificate (kubelet-client-ca-bundle)..."
level=debug msg="      Fetching Certificate (kube-control-plane-ca-bundle)..."
level=debug msg="        Fetching Certificate (kube-control-plane-signer)..."
level=debug msg="        Generating Certificate (kube-control-plane-signer)..."
level=debug msg="        Fetching Certificate (kube-apiserver-lb-signer)..."
level=debug msg="        Reusing previously-fetched Certificate (kube-apiserver-lb-signer)"
level=debug msg="        Fetching Certificate (kube-apiserver-localhost-signer)..."
level=debug msg="        Reusing previously-fetched Certificate (kube-apiserver-localhost-signer)"
level=debug msg="        Fetching Certificate (kube-apiserver-service-network-signer)..."
level=debug msg="        Reusing previously-fetched Certificate (kube-apiserver-service-network-signer)"
level=debug msg="      Generating Certificate (kube-control-plane-ca-bundle)..."
level=debug msg="      Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)..."
level=debug msg="        Fetching Certificate (kube-apiserver-to-kubelet-signer)..."
level=debug msg="        Generating Certificate (kube-apiserver-to-kubelet-signer)..."
level=debug msg="      Generating Certificate (kube-apiserver-to-kubelet-ca-bundle)..."
level=debug msg="      Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)..."
level=debug msg="        Fetching Certificate (kubelet-bootstrap-kubeconfig-signer)..."
level=debug msg="        Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-signer)"
level=debug msg="      Generating Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)..."
level=debug msg="    Generating Certificate (kube-apiserver-complete-client-ca-bundle)..."
level=debug msg="    Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-ca-bundle)"
level=debug msg="    Fetching Certificate (kube-apiserver-to-kubelet-client)..."
level=debug msg="      Fetching Certificate (kube-apiserver-to-kubelet-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer)"
level=debug msg="    Generating Certificate (kube-apiserver-to-kubelet-client)..."
level=debug msg="    Fetching Certificate (kube-apiserver-to-kubelet-signer)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer)"
level=debug msg="    Fetching Certificate (kube-control-plane-ca-bundle)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-control-plane-ca-bundle)"
level=debug msg="    Fetching Certificate (kube-control-plane-kube-controller-manager-client)..."
level=debug msg="      Fetching Certificate (kube-control-plane-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (kube-control-plane-signer)"
level=debug msg="    Generating Certificate (kube-control-plane-kube-controller-manager-client)..."
level=debug msg="    Fetching Certificate (kube-control-plane-kube-scheduler-client)..."
level=debug msg="      Fetching Certificate (kube-control-plane-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (kube-control-plane-signer)"
level=debug msg="    Generating Certificate (kube-control-plane-kube-scheduler-client)..."
level=debug msg="    Fetching Certificate (kube-control-plane-signer)..."
level=debug msg="    Reusing previously-fetched Certificate (kube-control-plane-signer)"
level=debug msg="    Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)..."
level=debug msg="    Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)"
level=debug msg="    Fetching Certificate (kubelet-client-ca-bundle)..."
level=debug msg="    Reusing previously-fetched Certificate (kubelet-client-ca-bundle)"
level=debug msg="    Fetching Certificate (kubelet-client)..."
level=debug msg="    Reusing previously-fetched Certificate (kubelet-client)"
level=debug msg="    Fetching Certificate (kubelet-signer)..."
level=debug msg="    Reusing previously-fetched Certificate (kubelet-signer)"
level=debug msg="    Fetching Certificate (kubelet-serving-ca-bundle)..."
level=debug msg="      Fetching Certificate (kubelet-signer)..."
level=debug msg="      Reusing previously-fetched Certificate (kubelet-signer)"
level=debug msg="    Generating Certificate (kubelet-serving-ca-bundle)..."
level=debug msg="    Fetching Certificate (mcs)..."
level=debug msg="    Reusing previously-fetched Certificate (mcs)"
level=debug msg="    Fetching Root CA..."
level=debug msg="    Reusing previously-fetched Root CA"
level=debug msg="    Fetching Key Pair (service-account.pub)..."
level=debug msg="    Generating Key Pair (service-account.pub)..."
level=debug msg="    Fetching Release Image Pull Spec..."
level=debug msg="    Generating Release Image Pull Spec..."
level=debug msg="Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:26760772b0128f11b3c3c13eeebf1238cf9b86ea5377808febdd14d4d00a2c31"
level=debug msg="    Fetching Image..."
level=debug msg="    Reusing previously-fetched Image"
level=debug msg="  Generating Bootstrap Ignition Config..."
level=debug msg="  Fetching Master Ignition Config..."
level=debug msg="  Reusing previously-fetched Master Ignition Config"
level=debug msg="  Fetching Master Machines..."
level=debug msg="  Reusing previously-fetched Master Machines"
level=debug msg="  Fetching Worker Machines..."
level=debug msg="  Reusing previously-fetched Worker Machines"
level=debug msg="Generating Terraform Variables..."
level=info msg="Consuming Install Config from target directory"
level=debug msg="Purging asset \"Install Config\" from disk"
level=debug msg="Fetching Kubeconfig Admin Client..."
level=debug msg="Loading Kubeconfig Admin Client..."
level=debug msg="  Loading Certificate (admin-kubeconfig-client)..."
level=debug msg="  Loading Certificate (kube-apiserver-complete-server-ca-bundle)..."
level=debug msg="  Loading Install Config..."
level=debug msg="  Fetching Certificate (admin-kubeconfig-client)..."
level=debug msg="  Reusing previously-fetched Certificate (admin-kubeconfig-client)"
level=debug msg="  Fetching Certificate (kube-apiserver-complete-server-ca-bundle)..."
level=debug msg="  Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle)"
level=debug msg="  Fetching Install Config..."
level=debug msg="  Reusing previously-fetched Install Config"
level=debug msg="Generating Kubeconfig Admin Client..."
level=debug msg="Fetching Kubeadmin Password..."
level=debug msg="Reusing previously-fetched Kubeadmin Password"
level=debug msg="Fetching Certificate (journal-gatewayd)..."
level=debug msg="Reusing previously-fetched Certificate (journal-gatewayd)"
level=debug msg="Fetching Metadata..."
level=debug msg="Loading Metadata..."
level=debug msg="  Loading Cluster ID..."
level=debug msg="  Loading Install Config..."
level=debug msg="  Fetching Cluster ID..."
level=debug msg="  Reusing previously-fetched Cluster ID"
level=debug msg="  Fetching Install Config..."
level=debug msg="  Reusing previously-fetched Install Config"
level=debug msg="Generating Metadata..."
level=debug msg="Fetching Cluster..."
level=debug msg="Loading Cluster..."
level=debug msg="  Loading Cluster ID..."
level=debug msg="  Loading Install Config..."
level=debug msg="  Loading Platform Credentials Check..."
level=debug msg="  Loading Platform Permissions Check..."
level=debug msg="    Loading Install Config..."
level=debug msg="  Loading Terraform Variables..."
level=debug msg="  Loading Kubeadmin Password..."
level=debug msg="  Fetching Cluster ID..."
level=debug msg="  Reusing previously-fetched Cluster ID"
level=debug msg="  Fetching Install Config..."
level=debug msg="  Reusing previously-fetched Install Config"
level=debug msg="  Fetching Platform Credentials Check..."
level=debug msg="  Reusing previously-fetched Platform Credentials Check"
level=debug msg="  Fetching Platform Permissions Check..."
level=debug msg="    Fetching Install Config..."
level=debug msg="    Reusing previously-fetched Install Config"
level=debug msg="  Generating Platform Permissions Check..."
level=fatal msg="failed to fetch Cluster: failed to fetch dependency of \"Cluster\": failed to generate asset \"Platform Permissions Check\": validate AWS credentials: mint credentials check: error simulating policy: Throttling: Rate exceeded\n\tstatus code: 400, request id: 1c8b583a-5f0f-42e3-afc3-02b168ca0ef9"


Expected results:
Installation to complete successfully

Additional info:
Running 4.4.6 instead, on the same environment and AWS account, is working as expected.

Comment 2 Scott Dodson 2020-06-23 19:50:55 UTC

Is this repeatable? Generally these errors are temporary and related to overall account utilization. The fact that you tried again with 4.4.6 and the problem went away lends itself to that conclusion as well.

Comment 3 Noam Manos 2020-06-24 08:01:05 UTC

This issue was not reproduced today (although it was reproduced 3 times during yesterday).

Comment 4 Brenton Leanhardt 2020-06-24 12:17:10 UTC

Let's close this for now.  It sounds like the quotas are indeed legitimately being hit.  AWS throttles their API on a per region basis so if you hit the problem and can try another region that would be one option forward.  The AWS documentation details how you can monitor throttling and even increase the quota for your account: 

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html

Comment 5 W. Trevor King 2020-06-24 15:09:29 UTC

Bug 1767936 landed a workaround for this in 4.5, but that workaround was not backported to 4.4.  Marking this as a dup to make it easier to discover the 4.5 workaround, but we can re-open this bug if we feel like it's worth backporting to 4.4.

*** This bug has been marked as a duplicate of bug 1767936 ***

Comment 6 W. Trevor King 2020-06-24 15:11:04 UTC

> AWS throttles their API on a per region basis so if you hit the problem and can try another region that would be one option forward.

I think the IAM endpoints used for permission checks are throttled per-account, so switching regions wouldn't help.  Not entirely sure though.