Description of problem: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-sdn-multitenant-4.2/124 level=info msg="Consuming \"Install Config\" from target directory" level=warning msg="Found override for ReleaseImage. Please be warned, this is not advised" level=info msg="Consuming \"Common Manifests\" from target directory" level=info msg="Consuming \"Worker Machines\" from target directory" level=info msg="Consuming \"Master Machines\" from target directory" level=info msg="Consuming \"Openshift Manifests\" from target directory" level=info msg="Creating infrastructure resources..." level=error level=error msg="Error: Error waiting for AMI (ami-0b59ba2db1a2d2747) to be ready: timeout while waiting for state to become 'available' (timeout: 40m0s)" level=error level=error msg=" on ../tmp/openshift-install-838476254/main.tf line 91, in resource \"aws_ami_copy\" \"main\":" level=error msg=" 91: resource \"aws_ami_copy\" \"main\" {" level=error level=error level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply using Terraform" Version-Release number of the following components: rpm -q openshift-ansible rpm -q ansible ansible --version How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag
> Error: Error waiting for AMI (ami-0b59ba2db1a2d2747) to be ready: timeout while waiting for state to become 'available' (timeout: 40m0s)" Seems like we waiting enough (40 mins) for the AMI copy to finish. Not sure the installer could have done anything.
About these slow AMI copies, here's what Darren L. at AWS told us on Fri Mar 22 2019, 16:48:00 GMT-0700: > 1. We have discussed with the internal team. Internally, we would try to address issues and shorten the duration of image creation/copy process, but however at the moment we do not have SLAs or official guidelines for expected copy times. > > 2. I have submitted a product feature request to our product development so that they could consider improving the image copy feature in the future.
It started impacting our CI a lot today: https://search.svc.ci.openshift.org/?search=Error+waiting+for+AMI&maxAge=6h&context=2&type=build-log Found 200 results in past 6 hours. It's the main contributor why AWS CI jobs fail. From https://prow.svc.ci.openshift.org/?job=*pull*aws* : Success rate over time: 3h: 6%, 12h: 26% We're 2 days before feature freeze and we cannot merge anything because of failed tests. I know it's mostly AWS fault, however, we should work around it somehow (not necessarily in the installer).
At a minimum perhaps we need to reduce our boskos lease quota so we have fewer jobs running simultaneously?
In our CI account, this can also manifest as: * AMI copies get slow. * We start failing out with the 40+m copies, closing the jobs and releasing the Boskos leases (making way for new jobs), but the slow copies were still running. * Copy Task limits also apply to same-region copies, which AWS does not currently document. They do have [1] Destination Regions are limited to 50 concurrent AMI copies. for cross-region copies, but the installer is always doing same-region copies. * We started hitting the Task limit between the currently-running jobs and the dangling slow copies from previous jobs. For example, [2]: level=error msg="Error: ResourceLimitExceeded: Task limit exceeded" level=error msg="\tstatus code: 400, request id: db3b367b-d2c0-4d1a-bec7-73a0c1b05dde" level=error msg="\tstatus code: 400, request id: db3b367b-d2c0-4d1a-bec7-73a0c1b05dde" level=error level=error msg=" on ../tmp/openshift-install-917883190/main.tf line 104, in resource \"aws_ami_copy\" \"main\":" level=error msg=" 104: resource \"aws_ami_copy\" \"main\" {" Updates from AWS: On Thu Jan 09 2020 14:42:05 GMT-0800, Derek H. wrote: > I have asked for our internal AMI team to confirm this is the case here and also to confirm the 50 limit applies for copies occuring within the same region. I will respond back once I have received more information from them. but no further details on that front yet. On Thu Jan 16 2020 16:02:09 GMT-0800, Derek H. wrote: > A few curiosities that popped up. There are ~5000 copies of snap-02ad473199123fb7a and snap-00196d4ecd27734ee each (and there are more such examples over just last one month). Can you explain why you would need to copy the same AMI/snapshot so many times? > > The problem seems to stem from multiple copy calls for the same resource. If you were copying a different resource each time we would expect to see less of an issue and faster completion. Please provide the use case and details so I can forward it to our internal team. From there they can evaluate or provide some suggestions on how we can improve the performance here. I explained that we use this account for OpenShift CI. Currently, creating a cluster involves copying official, unencrypted AMIs into the target AWS account so they can be encrypted. We can transition to launching encrypted-disk instances from the official AMI without the copy, but that's blocked on us bumping some internal Terraform [3]. Longer-term, we will probably transition to creating AMIs from VMDK instead of using official AMIs. But that means right now, and with our long-term approach, we are going to have lots of very similar AMIs being constantly created and destroyed in this account, because we spin up and tear down a lot of clusters. Most OpenShift consumers will likely spin up longer-running clusters, unless they are also performing CI on cluster creation and teardown. About the VMDK upload approach, e.g. the upload to S3 followed by an ImportImage call described in [4]. On Thu Jan 09 2020 15:41:57 GMT-0800, Derek H. wrote: > Checking here and it appears by default is a limit of 20 concurrent VM Import tasks. It does appear that this limit is not a hard limit and can be changed with approval from the VM Import team. [1]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/CopyingAMIs.html#copy-amis-across-regions [2]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_console-operator/370/pull-ci-openshift-console-operator-master-e2e-aws-operator/1577 [3]: https://github.com/openshift/installer/pull/2160#issuecomment-554239946 [4]: https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html#import-vm-image
> At a minimum perhaps we need to reduce our boskos lease quota so we have fewer jobs running simultaneously? I do not think this would help, because I expect the slow copies to be leaked out of aborted jobs. E.g. there is no 'Deleted ...' comment about the AMI getting deregistered [1] in [2]. It's possible that adjusting the installer to close the destroy-time leak of in-flight AMIs might be easier than bumping Terraform so we can drop the AMI copy entirely; I'm not sure. [1]: https://github.com/openshift/installer/blob/dd770f02b5263983fe04a41a2a4cb0669b4cf25e/pkg/destroy/aws/aws.go#L622 [2]: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_console-operator/370/pull-ci-openshift-console-operator-master-e2e-aws-operator/1577/artifacts/e2e-aws-operator/installer/.openshift_install.log
[1] landed a drop in the Boskos quota, so we'll see if that helps. [2] is in flight with some region-sharding. These are both just mitigations, so not linking formally from the issue. [1]: https://github.com/openshift/release/pull/6832 [2]: https://github.com/openshift/release/pull/6833
[1] is another example of the AMI copy error [1] https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-svcat-apiserver-operator/74/pull-ci-openshift-cluster-svcat-apiserver-operator-master-e2e-aws/210
Setting a target release so it doesn't show up as untriaged, although I don't think we need to deliver anything here for 4.4 if we can mitigate CI (e.g. via comment 7), because end users who aren't creating/destroying thousands of clusters a day in a single AWS account won't hit this (unless AWS breaks something else on their end).
Oh, and Boskos leases were dropped via [1], so we can clear the NEEDINFO from comment 4. [1]: https://github.com/openshift/release/pull/6832
This has not happened in the 24 hours since we landed the region sharding and lease reduction. Installer has a story to evaluate potential change that would reduce the overhead but unless we foresee customers continuously building and tearing down more than 50 concurrent clusters per region per account this is not likely to be something customers encounter. https://issues.redhat.com/browse/CORS-1352
LGTM. And from QE side, we did not hit such issue.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581